P-Tile Avalon Memory-Mapped IP For PCI Express User Guide

Intel® FPGA P-Tile Avalon® Memory-
mapped IP for PCI Express* User

Guide
Updated for Intel® Quartus® Prime Design Suite: 20.4
IP Version: 4.0.0
Subscribe UG-20237 | 2021.02.04

Send Feedback Latest document on the web: PDF | HTML
Contents
Contents
1. Introduction................................................................................................................... 4
1.1. Overview..............................................................................................................4
1.2. Features...............................................................................................................5
1.3. Release Information...............................................................................................6
1.4. Device Family Support............................................................................................7
1.5. Performance and Resource Utilization....................................................................... 7
1.6. IP Core and Design Example Support Levels..............................................................8
2. IP Architecture and Functional Description................................................................... 10
2.1. Top-Level Architecture.......................................................................................... 10
2.1.1. Avalon-MM Bridge Architecture.................................................................. 11
2.1.2. Clock Domains.........................................................................................13
2.1.3. Refclk.....................................................................................................15
2.1.4. Reset..................................................................................................... 17
2.2. Functional Description.......................................................................................... 18
2.2.1. PMA/PCS................................................................................................ 18
2.2.2. Data Link Layer Overview..........................................................................19
2.2.3. Transaction Layer Overview....................................................................... 21
2.2.4. Avalon-MM Bridge.................................................................................... 22
3. Parameters................................................................................................................... 26
3.1. Top-Level Settings............................................................................................... 26
3.2. Core Parameters.................................................................................................. 28
3.2.1. Base Address Registers.............................................................................28
3.2.2. PCI Express and PCI Capabilities Parameters............................................... 29
3.2.3. Device Identification Registers................................................................... 33
3.2.4. Configuration, Debug and Extension Options................................................34
3.3. Avalon-MM Settings..............................................................................................35
4. Interfaces..................................................................................................................... 37
4.1. Overview............................................................................................................ 37
4.2. Clocks and Resets................................................................................................ 38
4.2.1. Interface Clock Signals............................................................................. 38
4.2.2. Interface Reset Signals............................................................................. 39
4.3. Avalon-MM Interface ........................................................................................... 40
4.3.1. Endpoint Mode Interface (512-bit Avalon-MM Interface)................................ 42
4.3.2. Root Port Mode Interface (256-bit Avalon-MM Interface)............................... 58
4.4. Serial Data Interface............................................................................................ 61
4.5. Hard IP Status Interface....................................................................................... 61
4.6. Interrupt Interface............................................................................................... 62
4.6.1. Legacy Interrupts.................................................................................... 63
4.6.2. MSI........................................................................................................63
4.6.3. MSI-X.....................................................................................................66
4.7. Hot Plug Interface (RP Only)..................................................................................69
4.8. Power Management Interface................................................................................ 70
4.9. Configuration Output Interface.............................................................................. 71
4.10. Hard IP Reconfiguration Interface......................................................................... 75
4.10.1. Address Map for the User Avalon-MM Interface...........................................77
Intel® FPGA P-Tile Avalon® Memory-mapped IP for PCI Express* User Guide Send Feedback
2
Contents
4.10.2. Configuration Registers Access................................................................. 79

4.11. PHY Reconfiguration Interface..............................................................................81
5. Advanced Features....................................................................................................... 83
5.1. PCIe Port Bifurcation and PHY Channel Mapping....................................................... 83
6. Troubleshooting/Debugging......................................................................................... 84
6.1. Hardware............................................................................................................84
6.1.1. Debugging Link Training Issues.................................................................. 85
6.1.2. Debugging Data Transfer and Performance Issues........................................ 91
6.2. Debug Toolkit...................................................................................................... 94
6.2.1. Overview................................................................................................ 94
6.2.2. Enabling the P-Tile Debug Toolkit............................................................... 95
6.2.3. Launching the P-Tile Debug Toolkit............................................................. 95
6.2.4. Using the P-Tile Debug Toolkit....................................................................98
6.2.5. Enabling the P-Tile Link Inspector............................................................. 109
6.2.6. Using the P-Tile Link Inspector................................................................. 110
7. Document Revision History......................................................................................... 113
7.1. Document Revision History for the Intel FPGA P-Tile Avalon Memory-mapped IP for
PCI Express User Guide....................................................................................113
A. Configuration Space Registers.................................................................................... 115
A.1. Configuration Space Registers..............................................................................115
A.1.1. Register Access Definitions...................................................................... 117
A.1.2. PCIe Configuration Header Registers.........................................................117
A.1.3. PCI Express Capability Structures.............................................................119
A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure...............................121
A.1.5. MSI-X Registers..................................................................................... 121
A.2. Intel-Defined VSEC Capability Registers................................................................ 122
A.2.1. Intel-Defined VSEC Capability Header (Offset 00h)..................................... 123
A.2.2. Intel-Defined Vendor Specific Header (Offset 04h)...................................... 124
A.2.3. Intel Marker (Offset 08h)........................................................................ 124
A.2.4. JTAG Silicon ID (Offset 0x0C - 0x18)........................................................ 124
A.2.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D)....................... 124
A.2.6. General Purpose Control and Status Register (Offset 0x30)..........................125
A.2.7. Uncorrectable Internal Error Status Register (Offset 0x34)...........................125
A.2.8. Uncorrectable Internal Error Mask Register (Offset 0x38)............................ 126
A.2.9. Correctable Internal Error Status Register (Offset 0x3C)..............................127
A.2.10. Correctable Internal Error Mask Register (Offset 0x40).............................. 127
Send Feedback Intel® FPGA P-Tile Avalon® Memory-mapped IP for PCI Express* User Guide
3
UG-20237 | 2021.02.04
Send Feedback
1. Introduction
1.1. Overview
The P-Tile Avalon® memory mapped IP for PCIe combines the functionality of previous
Avalon memory-mapped (Avalon-MM) and Avalon memory-mapped with direct
memory access (DMA) interfaces. The IP core using the Avalon-MM interface removes
many of the complexities associated with the PCIe protocol. It handles all of the
Transaction Layer Packet (TLP) encoding and decoding, simplifying the design task. It
also includes optional Read and Write Data Mover modules facilitating the creation of
high-performance DMA designs. Both the Avalon-MM interface and the Read and Write
Data Mover modules are implemented in soft logic. This IP Core natively supports
Endpoint and Root Port configurations with Gen3/Gen4 data rates and x4/x8/x16 link
widths. Gen1/Gen2 data rates and x1/x2 link widths are supported via link down-
training.
The P-Tile Avalon memory mapped IP for PCIe consists of:

• Modules, implemented in soft logic, that perform Avalon memory mapped
functions. Together, these modules form an Avalon memory mapped Bridge.
• A PCIe Hard IP that implements the Transaction, Data Link, and Physical layers
stack that is compliant with PCI Express Base Specification 4.0 . This stack allows
the user application logic in the Intel FPGA to interface with another device via a
PCI Express link.
This IP provides support for an Avalon memory mapped interface with DMA and is
designed to optimize the performance of large-size data transfers. If you want to
achieve maximum performance with small-size transfers, Intel recommends the use of
the P-Tile Avalon streaming IP for PCIe.
Note: The P-Tile Avalon memory mapped IP for PCIe does not include an internal descriptor
controller for DMA operations. This descriptor controller should be implemented in the
user application logic. The design example provided for this IP includes an example of
a descriptor controller.
Related Information
Intel FPGA P-Tile Avalon memory mapped IP for PCI Express Design Example User
Guide
Intel Corporation. All rights reserved. Agilex, Altera, Arria, Cyclone, Enpirion, Intel, the Intel logo, MAX, Nios,
Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or
other countries. Intel warrants performance of its FPGA and semiconductor products to current specifications in ISO
accordance with Intel's standard warranty, but reserves the right to make changes to any products and services 9001:2015
at any time without notice. Intel assumes no responsibility or liability arising out of the application or use of any Registered
information, product, or service described herein except as expressly agreed to in writing by Intel. Intel
customers are advised to obtain the latest version of device specifications before relying on any published
information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
1. Introduction
UG-20237 | 2021.02.04
1.2. Features
The P-Tile Avalon memory mapped IP for PCI Express supports the following features:
• Configurations supported:
Table 1. Configurations Supported by the P-Tile Avalon memory mapped IP for PCI
Express
Gen3/Gen4 x16 Gen3/Gen4 x8 Gen3/Gen4 x4
Endpoint (EP) Yes Yes N/A
Root Port (RP) (1) N/A Yes
Note: Gen1/Gen2 configurations are supported via link down-training.

• Support for 256-bit and 512-bit data paths.
• 512-bit data path with 250 MHz interfaces to user logic to ease timing closure for
Gen3 x16.
• Support for a single function (PF0).
• High-throughput Bursting Avalon memory mapped Slave (BAS).
— Byte enables with byte granularity.
• High-throughput Bursting Avalon memory mapped Master (BAM).
• Support for up to 7 BARs, including expansion ROM BAR.
• Support for byte enables with byte granularity.
• Support for up to 64 outstanding Non-Posted requests.
• Summary of outstanding Non-Posted requests supported:
Table 2. Outstanding Non-Posted Requests Supported

Ports Active Cores Outstanding Non-Posted Requests
0 x16 64, 512 (*)
1 x8 64
2 and 3 x4 64
Note: (*) : 512 outstanding Non-Posted requests support may be available in a

future Intel Quartus Prime release.
• Data movers with high throughput for DMA support
— Move data using PCIe Memory Read and Memory Write packets.
— Bursting Avalon memory mapped Master interfaces for data path.
— Byte enables with dword granularity.
— Avalon streaming interfaces for control and status.
— DMA transfers of 1 dword to (1 MB - 1 dword) in 1 dword increments.
— All addresses are dword-aligned.
(1) These configurations may be available in a future release of Intel® Quartus® Prime.
5
1. Introduction
UG-20237 | 2021.02.04
• Bursts of up to 8 cycles (512 bytes) for the Bursting Avalon memory mapped
Master, Bursting Avalon memory mapped Slave and the data movers.
• Support for Max Payload Size values of 128, 256 and 512 bytes.
• Support for Max Read Request Size values of 128, 256 and 512 bytes.
• Available as a Platform Designer component with standard Avalon interfaces.
• MSI and MSI-X.
• Separate Refclk with Independent Spread Spectrum Clocking (SRIS).
• You cannot change the pin allocations for the P-Tile Avalon memory mapped IP for
PCI Express* in the Intel Quartus Prime project. However, this IP does support
lane reversal and polarity inversion on the PCB.
• Supports Autonomous Hard IP mode.
— This mode allows the PCIe Hard IP to communicate with the Host before the
FPGA configuration and entry into User mode are complete.
Note: Unless Readiness Notifications mechanisms are used, the Root Complex
and/or system software must allow at least 1.0 s after a Conventional
Reset of a device before it may determine that a device that fails to
return a Successful Completion status for a valid Configuration Request
is a broken device. This period is independent of how quickly Link
training completes.
• Modular implementation allowing users to enable the required features for a
specific application. For example:
— Simultaneous support for DMA modules and high-throughput Avalon memory
mapped Slaves and Masters.
— Avalon memory mapped Slave for easy access to the whole PCIe address
space.
• VCS is the only simulator supported in the 20.2 release of Intel Quartus Prime.
Other simulators may be supported in a future release.
Note: Throughout this User Guide, the term Avalon-MM may be used as an abbreviation for
the Avalon memory mapped interface or IP.
1.3. Release Information

Table 3. P-Tile Avalon memory mapped IP for PCI Express Release Information
Item Description
IP Version 4.0.0
Intel Quartus Prime Version 20.4
Release Date December 2020
Ordering Codes No ordering code is required
IP versions are the same as the Intel Quartus Prime Design Suite software versions up
to v19.1. From Intel Quartus Prime Design Suite software version 19.2 or later, IPs
have a new IP versioning scheme.
The IP version (X.Y.Z) number may change from one Intel Quartus Prime software
version to another. A change in:
6
1. Introduction
UG-20237 | 2021.02.04
• X indicates a major revision of the IP. If you update your Intel Quartus Prime
software, you must regenerate the IP.
• Y indicates the IP includes new features. Regenerate your IP to include these new
features.
• Z indicates the IP includes minor changes. Regenerate your IP to include these
changes.
Intel verifies that the current version of the Intel Quartus Prime Pro Edition software
compiles the previous version of each IP core, if this IP core was included in the
previous release. Intel reports any exceptions to this verification in the Intel IP
Release Notes or clarifies them in the Intel Quartus Prime Pro Edition IP Update tool.
Intel does not verify compilation with IP core versions older than the previous release.
Related Information
P-Tile IP for PCI Express IP Core Release Notes
This document provides information about the new features and updates for each
IP release.
1.4. Device Family Support

The following terms define device support levels for Intel FPGA IP cores:
• Advance support—the IP core is available for simulation and compilation for this
device family. Timing models include initial engineering estimates of delays based
on early post-layout information. The timing models are subject to change as
silicon testing improves the correlation between the actual silicon and the timing
models. You can use this IP core for system architecture and resource utilization
studies, simulation, pinout, system latency assessments, basic timing assessments
(pipeline budgeting), and I/O transfer strategy (data-path width, burst depth, I/O
standards tradeoffs).
• Preliminary support—the IP core is verified with preliminary timing models for
this device family. The IP core meets all functional requirements, but might still be
undergoing timing analysis for the device family. It can be used in production
designs with caution.
• Final support—the IP core is verified with final timing models for this device
family. The IP core meets all functional and timing requirements for the device
family and can be used in production designs.
Table 4. Device Family Support

Device Family Support Level
Intel Stratix® 10 DX Preliminary support
Intel Agilex™ Preliminary support
No support
Other device families Refer to the Intel PCI Express Solutions web page on the Intel website for support information on
other device families.
1.5. Performance and Resource Utilization

The following table shows the recommended FPGA fabric speed grades for all the
configurations that the Avalon-MM IP core supports.
7
1. Introduction
UG-20237 | 2021.02.04
Table 5. Intel Stratix 10 DX / Intel Agilex Recommended FPGA Fabric Speed Grades
for All Avalon-MM Widths and Frequencies
The recommended FPGA fabric speed grades are for production parts.
Lane Rate Link Width Application Interface Application Clock Recommended FPGA
Data Width Frequency (MHz) Fabric Speed Grades
200 MHz (Intel Stratix

x4 256-bit 10 DX) -1, -2
250 MHz (Intel Agilex)

x8 512-bit 10 DX) -1, -2
Gen4
250 MHz (Intel Agilex)

10 DX)
x16 512-bit -1, -2
350 MHz / 400 MHz
(Intel Agilex)
x4 256-bit 125 MHz -1, -2
Gen3 x8 512-bit 125 MHz -1, -2
x16 512-bit 250 MHz -1, -2
The Avalon-MM variants include an Avalon-MM DMA bridge implemented in soft logic.
It operates as a front end to the hardened protocol stack. The resource utilization
table below shows results for the Simple DMA dynamically generated design example.
The results are for the current version of the Intel Quartus Prime Pro Edition software.
Table 6. Resource Utilization of the Avalon-MM IP for PCI Express IP Core

Design Link Device Family Typical ALMs M20K Memory Logic Registers
Example Used Configuration Blocks(2)
Intel Stratix 10
DMA Gen3 x16, EP 15956 120 42345
DX
DMA Gen3 x16, EP Intel Agilex 17116 120 42940
Intel Stratix 10
DMA Gen4 x16, EP 15967 120 42641
DX
Intel Stratix 10
DMA Gen4 x8, EP 14533 97 42610
DX
1.6. IP Core and Design Example Support Levels

The following table shows the support levels of the Avalon-MM IP core and design
example in Intel Stratix 10 DX devices.
(2) These results only include resources in the Avalon-MM IP partition in the design example as
the Table title indicates. They do not include resources for external blocks such as the on-chip
memory, DMA controller, and other interconnect logic.
8
1. Introduction
UG-20237 | 2021.02.04
Table 7. P-Tile Avalon Memory Mapped (Avalon-MM) IP for PCIe Support Matrix for
Intel Stratix 10 DX Devices
Support level keys: S = simulation, C = compilation, T = timing, H = hardware, N/A = configuration not
supported
PCIe IP Support Design Example Support

Configuration
EP RP EP RP
Gen4 x16 512-bit SCTH (††) S C T H (†) (††)
Gen4 x8/x8 512-bit SCTH N/A S C T H (†) N/A
Gen4 x4/x4/x4/x4
N/A SCTH N/A (††)
256-bit
Gen3 x4/x4/x4/x4
256-bit
Note: (†) The design example available in the 20.4 release supports the DMA mode with
Data Movers. A design example supporting the Bursting Slave mode may be available
in a future release.
Note: (††) This support may be available in a future release of Intel Quartus Prime.
The following table shows the support levels of the Avalon-MM IP core and design
example in Intel Agilex devices.
Table 8. P-Tile Avalon Memory Mapped (Avalon-MM) IP for PCIe Support Matrix for
Intel Agilex Devices
Support level keys: S = simulation, C = compilation, T = timing, H = hardware, N/A = configuration not
supported
PCIe IP Support Design Example Support

Configuration
EP RP EP RP
Gen4 x4/x4/x4/x4
256-bit
Gen3 x4/x4/x4/x4
256-bit
Note: (†) The design example available in the 20.4 release supports the DMA mode with
Data Movers. A design example supporting the Bursting Slave mode may be available
in a future release.
Note: (††) This support may be available in a future release of Intel Quartus Prime.
9
UG-20237 | 2021.02.04
Send Feedback
2. IP Architecture and Functional Description
2.1. Top-Level Architecture

The P-tile Avalon-MM IP for PCI Express consists of the following major sub-blocks:
• PMA/PCS
• Four PCIe* cores (one x16 core, one x8 core and two x4 cores)
• Embedded Multi-die Interconnect Bridge (EMIB)
• Soft logic blocks in the FPGA fabric to implement the Avalon-MM Bridge, which
translates the PCIe TLPs from the PCIe Hard IP into standard Avalon memory-
mapped reads and writes.
Figure 1. P-tile Avalon-MM IP for PCI Express top-level block diagram

P-Tile Avalon-MM PCIe IP Top Level
P-Tile EMIB FPGA Fabric

PHY
PMA Quad 3 PCIe Controllers
PLL A/B x4
PHY Datax4 Trans-
PMA Quad 2 Layer
PHY Link Datax8action
Trans-
Bifurcation Mux
PLL A/B (MAC)

LayerPHYLayer Layer
LinkData action
PCIe x16 Trans- Avalon
PCIe (MAC) x16
LayerLink Layer User
x16 Lanes Layer action -MM
PMA Quad 1 PCS (MAC)PHY Data Trans- Bridge Logic
Layer Layer Link Layeraction
PLL A/B (MAC) Layer Layer
refclk0
refclk1 PMA Quad 0
PLL A/B
pin_perst_n
Note: Each core in the IP implements its own Data Link Layer and Transaction Layer.
The four cores in the IP can be configured to support the following topologies:
Table 9. Configuration Modes Supported by the P-tile Avalon-MM IP for PCI Express
Endpoint
Configuration Mode Native Hard IP Mode (EP) / Root Active Cores
Port (RP)
Configuration Mode 0 Gen3x16 or Gen4x16 EP/RP x16
Configuration Mode 1 Gen3x8/Gen3x8 or Gen4x8/Gen4x8 EP x16, x8
Gen3x4/Gen3x4/Gen3x4/Gen3x4 or
Configuration Mode 2 RP x16, x8, x4_0, x4_1
Gen4x4/Gen4x4/Gen4x4/Gen4x4
UG-20237 | 2021.02.04
In Configuration Mode 0, only the x16 core is active, and it operates in Gen3 x16
mode or Gen4 x16 mode.
In Configuration Mode 1, the x16 core and x8 core are active, and they operate as two
Gen3 x8 cores or two Gen4 x8 cores.
In Configuration Mode 2, all four cores (x16, x8, x4_0, x4_1) are active, and they
operate as four Gen3 x4 cores or four Gen4 x4 cores.
2.1.1. Avalon-MM Bridge Architecture

The P-Tile Avalon-MM Bridge can support three modes of operation:
• Endpoint mode with Data Movers.
• Endpoint mode.
• Root Port mode.
In the first two modes, the P-Tile Avalon-MM IP functions as an Endpoint (EP). In Root
Port mode, it functions as a Root Port (RP).
The Avalon-MM Bridge consists of five main modules: Read Data Mover (RDDM), Write
Data Mover (WRDM), Bursting Avalon-MM Master (BAM), Bursting Avalon-MM Slave
(BAS) and Control Register Access (CRA). These modules are shown in Figure 2 on
page 12 and described below. Depending on the mode of operation, different
modules in the IP core are enabled.
Table 10. Operating Modes of the Avalon-MM Bridge

In the following table, Yes means the block is enabled for that operating mode. No means the block is not
enabled for that mode.
Modules
Bursting Avalon-MM Bursting Avalon-MM

Master (BAM) Slave (BAS)
Modes Read Data Write Data
Control Register
Mover Mover
Non- Non- Access (CRA)
(RDDM) (WRDM) Bursting Bursting
Bursting Bursting
Mode Mode
Mode Mode
Endpoint
mode with
Yes Yes Yes No No No No
Data Movers
(EP)
Endpoint
No No Yes No No Yes No
mode (EP)
Root Port
No No No Yes Yes No Yes
mode (RP)
11
UG-20237 | 2021.02.04
Here is a block diagram of the P-Tile Avalon-MM Bridge showing the main modules:
Figure 2. P-Tile Avalon-MM Bridge Block Diagram
P-Tile Avalon-MM IP for PCIe M Avalon-MM Master I Avalon-ST Sink C Avalon Conduit
S Avalon-MM Slave O Avalon-ST Source
250 MHz
Avalon-MM Bridge
500 MHz
Control Register O
S Access
I
I
I
Write
O Data Mover O
M
512 512
I Read
Embedded/Separated
I O
Hard IP Interface
Data Mover
Header Adaptor
O
Width and Rate Adaptor

M I
P-Tile PCle Avalon-ST

Response
P-Tile PCle Hard IP

Re-ordering
Bursting O
S Slave O
I
Bursting O
512 512
M Master
I
12
UG-20237 | 2021.02.04
• Bursting Master (BAM): This module converts memory read and write TLPs
initiated by the remote link partner and received over the PCIe link into Avalon-
MM burst read and write transactions, and sends back CplD TLPs for read requests
it receives. It can also function in a non-bursting mode.
• Bursting Slave (BAS): This module converts Avalon-MM read and write
transactions initiated by the application logic into PCIe memory read and write
TLPs to be transmitted over the PCIe link. This module also processes the CplD
TLPs received for the read requests it sent. It can also function in a non-bursting
mode.
• Read Data Mover (RDDM): This module uses PCIe memory read TLPs and Avalon-
MM write transactions to move large amounts of data from the system memory in
the PCIe space to the FPGA memory in the Avalon-MM space. It fetches
descriptors from the system memory through one of its two Avalon-ST sink
interfaces. These descriptors define the data transfers to be executed. The RDDM
also reports the status of these data transfers via its Avalon-ST source interface.
• Write Data Mover (WRDM): This module uses PCIe memory write TLPs and
Avalon-MM read transactions to move large amounts of data from your application
logic in the Avalon-MM space to the system memory in the PCIe space. The WRDM
also supports immediate writes, which are enabled by a bit in the descriptors that
the WRDM receives via one of its Avalon-ST descriptor sink interfaces. For more
details on immediate writes, refer to Write Data Mover Avalon-ST Descriptor Sinks
on page 52. Similar to the RDDM, the WRDM also has its own Avalon-ST source
interface to report the status of its data transfers.
• Control Register Access (CRA) Avalon-MM Slave (Root Port only): This module is
used in Root Port mode only to issue accesses to the Endpoint's configuration
space registers. It supports a single transaction at a time. It converts single-cycle,
32-bit Avalon-MM read and write transactions into PCIe configuration read and
write TLPs (CfgRd0, CfgRd1, CfgWr0 and CfgWr1) to be sent over the PCIe link.
This module also processes the completion TLPs (Cpl and CplD) it receives in
return.
The Response Reordering module assembles and reorders completion TLPs received
over the PCIe link for the Bursting Slave and the Read Data Mover. It routes the
completions based on their tags.
No re-ordering is necessary for the completions sent to the CRA module as it only
issues one request TLP at a time.
Endpoint applications typically need the Bursting Master to enable the host to provide
information for the other modules.
2.1.2. Clock Domains

The P-Tile Avalon-MM IP for PCI Express has three primary clock domains:
13
UG-20237 | 2021.02.04
• PHY clock domain (i.e. core_clk domain): this clock is synchronous to the
SerDes parallel clock.
• EMIB/FPGA fabric interface clock domain (i.e. pld_clk domain): this clock is
derived from the same reference clock (refclk0) as the one used by the SerDes.
However, this clock is generated from a stand-alone core PLL.
• Application clock domain (p<n>_app_clk): this clock is an output from the P-Tile
IP. The frequency of this clock depends on the configuration that the IP is in. Refer
to Table 11 on page 15 below for more details. This is a per-port signal (i.e, n =
0,1,2,3).
Figure 3. Clock Domains
FPGA Fabric P-Tile x16 core_clk

x8 core_clk
x4_0 core_clk
x4_1 core_clk
Avalon- P P
User
MM EMIB PCIe Hard IP C M pld_clk
Logic
Bridge S A p<n>_app_clk
The PHY clock domain (i.e. core_clk domain) is a dynamic frequency domain. The
PHY clock frequency is dependent on the current link speed.
14
UG-20237 | 2021.02.04
Table 11. PHY Clock and Application Clock Frequencies

Link Speed PHY Clock Frequency Application Clock Frequency
Gen1 is supported only via link down-training and not natively. Hence,
Gen1 125 MHz the application clock frequency depends on the configuration you
choose in the IP Parameter Editor.
Gen2 is supported only via link down-training and not natively. Hence,
Gen2 250 MHz the application clock frequency depends on the configuration you
choose in the IP Parameter Editor.
p<n>_app_clk Frequency
(MHz)
Configurati Data Width
EP/RP
on (bits) Intel
Stratix 10 Intel Agilex
DX
Gen3 500 MHz
Gen3 x4 RP 256 125 125
Gen3 x8 EP 512 125 125
Gen3 x16 EP 512 250 250
p<n>_app_clk Frequency
(MHz)
Configurati Data Width
EP/RP
on (bits) Intel
Stratix 10 Intel Agilex
DX
Gen4 1000 MHz
Gen4 x4 RP 256 200 250
Gen4 x8 EP 512 200 250
Gen4 x16 EP 512 350 400
Note: Refer to the P-Tile IP for PCI Express IP Core Release Notes for the matrix of
configurations supported by the P-Tile Avalon memory mapped IP for PCI Express.
2.1.3. Refclk
P-Tile has two reference clock inputs at the package level, refclk0 and refclk1.
You must connect a 100 MHz reference clock source to these two inputs. Depending
on the port mode, you can drive the two refclk inputs using either a single clock
source or two independent clock sources.
In 1x16 and 4x4 modes, drive the refclk inputs with a single clock source (through
a fanout buffer) as shown in the figure below.
15
UG-20237 | 2021.02.04
Figure 4. Using a Single 100 MHz Clock Source in 1x16 and 4x4 Modes
PCIe x4 PCIe x8/ PCIe x16/ PCIe x4

x4 x8/x4
(Port 3) (Port 1) (Port 0) (Port 2)
PHY3 PHY2 PHY1 PHY0

Lane 15
Lane 12
Lane 11
Lane 8
Lane 3
Lane 0
Lane 7
Lane 4
(x4) (x4) (x4) (x4)
Refclk distribution on the package substrate
Refclk1 Refclk0
Fanout Buffer
100MHz
In 2x8 mode, you can drive the refclk inputs with either a single 100 MHz clock
source as shown above, or two independent 100 MHz sources (see the figure below)
depending on your system architecture. For example, if your system has each x8 port
connected to a separate CPU/Root Complex, it may be required to drive these refclk
inputs using independent clock sources. In that case, it is strongly recommended that
the refclk0 input for Port 0 (lanes 0 - 7) be always running because it feeds the
reference clock for the P-Tile core PLL that controls the data transfers between the P-
Tile and FPGA fabric via the EMIB. If this clock goes down, Port 0 link will go down and
Port 1 will not be able to communicate with the FPGA fabric. Following are the
guidelines for implementing two independent refclks in 2x8 mode:
• If the link can handle two separate reference clocks, drive the refclk0 of P-Tile
with the on-board free-running oscillator.
• If the link needs to use a common reference clock, then PERST# needs to indicate
the stability of this reference clock. If this reference clock goes down, the entire P-
Tile must be reset.
16
UG-20237 | 2021.02.04
Figure 5. Using Independent 100 MHz Clock Sources in 2x8 Mode
PCIe x4 PCIe x8/ PCIe x16/ PCIe x4

x4 x8/x4
(Port 3) (Port 1) (Port 0) (Port 2)
Lane 15 PHY3 PHY2 PHY1 PHY0
Lane 12
Lane 11
Lane 8
Lane 3
Lane 0
Lane 7
Lane 4
(x4) (x4) (x4) (x4)
Refclk distribution on the package substrate
Refclk1 Refclk0
100MHz 100MHz
2.1.4. Reset
There is only one PERST# (pin_perst_n) pin on P-Tile. Therefore, toggling
pin_perst_n will affect the entire P-Tile. If the P-Tile x16 port is bifurcated into two
x8 Endpoints, toggling pin_perst_n will affect both x8 Endpoints.
To reset each port individually, use the in-band mechanism like Hot Reset.
Following are the guidelines for implementing the P-Tile pin_perst_n reset signal:
• pin_perst_n is a "power good" indicator from the associated power domain (to
which P-Tile is connected). Also, it shall qualify that both the P-Tile refclk0 and
refclk1 are stable. If one of the reference clocks becomes stable later, deassert
pin_perst_n after this reference clock becomes stable.
• pin_perst_n assertion is required for proper Autonomous P-Tile functionality. In
Autonomous mode, P-Tile can successfully link up upon the release of
pin_perst_n regardless of the FPGA fabric configuration and will send out CRS
(Configuration Retry Status) until the FPGA fabric is configured and ready.
The following is an example where a single PERST# (pin_perst_n) is driven with

independent refclk0 and refclk1. In this example, the add-in card (FPGA and Soc)
is powered up first. P-Tile refclk0 is fed by the on-board free-running oscillator. P-
Tile refclk1 driven by the Host becomes stable later. Hence, the PERST# is
connected to the Host.
17
UG-20237 | 2021.02.04
Figure 6. Single PERST# Connection in Bifurcated 2x8 Mode
Host System Add In FPGA Card

(RP#0)
100 MHz
Oscillator
refclk1
FPGA SoC
pin_perst_n (P-tile) refclk0 (RP#1)
2.2. Functional Description
2.2.1. PMA/PCS
The P-Tile Avalon-MM IP for PCI Express contains Physical Medium Attachment (PMA)
and PCI Express Physical Coding Sublayer (PCIe PCS) blocks for handling the Physical
layer (PHY) packets. The PMA receives and transmits high-speed serial data on the
serial lanes. The PCS acts as an interface between the PMA and the PCIe controller,
and performs functions like data encoding and decoding, scrambling and
descrambling, block synchronization etc. The PCIe PCS in the P-Tile Avalon-MM IP for
PCI Express is based on the PHY Interface for PCI Express (PIPE) Base Specification
4.4.1.
In this IP, the PMA consists of up to four quads. Each quad contains a pair of transmit
PLLs and four SerDes lanes capable of running up to 16 GT/s to perform the various
TX and RX functions.
PLLA generates the required transmit clocks for Gen1/Gen2 speeds, while PLLB
generates the required clocks for Gen3/Gen4 speeds. For the x8 and x16 lane widths,
one of the quads acts as the master PLL source to drive the clock inputs for each of
the lanes in the other quads.
The PMA performs functions such as serialization/deserialization, clock data recovery,

and analog front-end functions such as Continuous Time Linear Equalizer (CTLE),
Decision Feedback Equalizer (DFE) and transmit equalization.
The transmitter consists of a 3-tap equalizer with one tap of pre-cursor, one tap of
main cursor and one tap of post-cursor.
The receiver consists of attenuation (ATT), CTLE, Voltage gain amplifier (VGA) and a
5-tap DFE blocks that are adaptive for Gen3/Gen4 speeds. RX Lane Margining is
supported by the PHY. The Lane Margining supports timing margining only. The
optional voltage margining is not supported. Timing margining capabilities/parameters
are as follows:
18
UG-20237 | 2021.02.04
• Maximum Timing Offset: -0.2UI to +0.2UI.

• Number of timing steps in each direction: 9.
• Independent left and right timing margining is supported.
• Independent Error Sampler is not supported (lane margining may produce logical
errors in the data stream and cause the LTSSM to go to the Recovery state).
The PHY layer uses a fixed 16-bit PCS-PMA interface width to output the PHY clock
(core_clk). The frequency of this clock is dependent on the current link speed. Refer
to Table 11 on page 15 for the frequencies at various link speeds.
Related Information
PHY Interface for PCI Express (PIPE) Base Specification
2.2.2. Data Link Layer Overview

The Data Link Layer (DLL) is located between the Transaction Layer and the Physical
Layer. It maintains packet integrity and communicates (by DLL packet transmission) at
the PCI Express link level.
The DLL implements the following functions:

• Link management through the reception and transmission of DLL Packets (DLLP),
which are used for the following functions:
— Power management of DLLP reception and transmission
— To transmit and receive ACK/NAK packets
— Data integrity through the generation and checking of CRCs for TLPs and
DLLPs
— TLP retransmission in case of NAK DLLP reception or replay timeout, using the
retry (replay) buffer
— Management of the retry buffer
— Link retraining requests in case of error through the Link Training and Status
State Machine (LTSSM) of the Physical Layer
19
UG-20237 | 2021.02.04
Figure 7. Data Link Layer

To Transaction Layer To Physical Layer
Tx Transaction Layer Tx Arbitration

Packet Description & Data Transaction Layer
Packet Generator Tx Packets
Retry Buffer DLLP TX Datapath

Generator
Ack/Nack
Packets
Data Link Control Control
Power and Management & Status
Configuration Space
Management State Machine
Tx Flow Control Credit Information Function
Rx Flow Control Credit Information DLLP RX Datapath

Checker
Transaction Layer
Packet Checker Rx Packets
Rx Transation Layer
Packet Description & Data
Note:
The DLL has
(1) The L0s the
(Standby) or L1following sub-blocks:
(Low Power Standby) states are not supported.
• Data Link Control and Management State Machine—This state machine connects to
both the Physical Layer’s LTSSM state machine and the Transaction Layer. It
initializes the link and flow control credits and reports status to the Transaction
Layer.
• Power Management—This function handles the handshake to enter low power
mode. Such a transition is based on register values in the Configuration Space and
received Power Management (PM) DLLPs. For more details on the power states
supported by the P-Tile Avalon-MM IP for PCIe, refer to section Power Management
Interface on page 70.
• Data Link Layer Packet Generator and Checker—This block is associated with the
DLLP’s 16-bit CRC and maintains the integrity of transmitted packets.
• Transaction Layer Packet Generator—This block generates transmit packets,
including a sequence number and a 32-bit Link CRC (LCRC). The packets are also
sent to the retry buffer for internal storage. In retry mode, the TLP generator
receives the packets from the retry buffer and generates the CRC for the transmit
packet.
• Retry Buffer—The retry buffer stores TLPs and retransmits all unacknowledged
packets in the case of NAK DLLP reception. In case of ACK DLLP reception, the
retry buffer discards all acknowledged packets.
20
UG-20237 | 2021.02.04
• ACK/NAK Packets—The ACK/NAK block handles ACK/NAK DLLPs and generates the
sequence number of transmitted packets.
• Transaction Layer Packet Checker—This block checks the integrity of the received
TLP and generates a request for transmission of an ACK/NAK DLLP.
• TX Arbitration—This block arbitrates transactions, prioritizing in the following
order:
— Initialize FC Data Link Layer packet
— ACK/NAK DLLP (high priority)
— Update FC DLLP (high priority)
— PM DLLP
— Retry buffer TLP
— TLP
— Update FC DLLP (low priority)
— ACK/NAK FC DLLP (low priority)
2.2.3. Transaction Layer Overview

The following figure shows the major blocks in the P-Tile Avalon-MM IP for PCI Express
Transaction Layer:
Figure 8. P-Tile Avalon-MM IP for PCI Express Transaction Layer Block Diagram
Avalon-MM Avalon-ST
RX RX
RX
User Avalon-MM
RAS
Data Link Layer
Avalon-MM Bridge
+
CONFIG CPL Timeout
Logic RAS Physical Layer
Avalon-MM Avalon-ST
TX TX
TX
The RAS (Reliability, Availability, and Serviceability) block includes a set of features to
maintain the integrity of the link.
For example: Transaction Layer inserts an optional ECRC in the transmit logic and
checks it in the receive logic to provide End-to-End data protection.
When the application logic sets the TLP Digest (TD) bit in the Header of the TLP, the P-
Tile Avalon-MM IP for PCIe will append the ECRC automatically.
The TX block sends out the TLPs that it receives as-is. It also sends the information
about non-posted TLPs to the Completion (CPL) Timeout Block for CPL timeout
detection.
The P-Tile Avalon-MM IP for PCI Express RX block consists of two main blocks:
21
UG-20237 | 2021.02.04
• Filtering block: This module checks if the TLP is good or bad and generates the
associated error message and completion. It also tracks received completions and
updates the completion timeout (CPL timeout) block.
• RX Buffer Queue: The P-Tile IP for PCIe has separate queues for posted/non-
posted transactions and completions. This avoids head-of-queue blocking on the
received TLPs and provides flexibility to extract TLPs according to the PCIe
ordering rules.
Figure 9. P-Tile Avalon-MM IP for PCI Express RX Block Overview
Trash RX Buffer Queue Filter
TLP Filtering
P
Avalon Width Received CPL

Avalon-MM Avalon-ST and
-MM Avalon-ST Avalon-ST Routing NP Processing (*)
Bridge Rate Data Logical
Adapter
Message Link PHY
CPL Processing Layer Layer
MSG
ERR
MSG
User Avalon-MM Config CFG Data TX
Note: The Received CPL Processing block includes the CPL tracking mechanism.
2.2.4. Avalon-MM Bridge

As described in Table 10 on page 11, the P-Tile Avalon-MM Bridge can support three
modes of operation:
• Endpoint mode with Data Movers.
• Endpoint mode.
• Root Port mode.
2.2.4.1. Endpoint Mode with Data Movers
When the Avalon-MM Bridge is used in this mode, the following modules are enabled:
• Read Data Mover (RDDM)
• Write Data Mover (WRDM)
• Bursting Master (BAM) in Non-Bursting Mode
The following figure shows how the DMA example design that you can generate using
the Intel Quartus Prime software interfaces with the P-Tile Avalon-MM IP to perform
DMA operations. If you are not using the provided DMA example design, you need to
implement your custom DMA Controller and BAR Interpreter in your application logic.
22
UG-20237 | 2021.02.04
Figure 10. P-Tile Avalon-MM IP in Endpoint Mode with Data Movers Enabled
DMA Example Design S P-Tile Avalon-MM IP ASTI Avalon-ST Sink M Avalon-MM Master
Memory S
ASTO Avalon-ST Source S Avalon-MM Slave
C Avalon Conduit
Qsys Interconnect ASTI
WASTO
ASTI
Write
WASTO data mover ASTO
WASTI ASTO
DMA M (512 bits) 512
S TX
Controller
ASTI Read
RASTO ASTO
ASTI
RASTO
ASTO
data mover P-Tile
RASTI ASTI
M (512 bits) Avalon-ST PCIe
Completion Interface Hard IP
Re-ordering
RX 512
C Bursting Master
BAR C M (non-bursting ASTO
M S mode) ASTI
Interpreter (512 bits)
In this DMA example design, the BAM is used in non-bursting mode by the host to
program the Control and Status registers of the DMA controller in the user Avalon-MM
space. The DMA controller, after being programmed, sends descriptor-fetching
instructions to the host via the RDDM. After the fetched descriptors are processed by
the WRDM and RDDM, status and/or MSI-X messages are sent to the host via the
WRDM in “Immediate” mode. In this mode, the data payload is embedded in bits
[31:0] or [63:0] of the fetched descriptors that the WRDM receives (depending on
whether a one- or two-dword immediate transfer is needed respectively). For more
details on immediate transfers, refer to Write Data Mover Avalon-ST Descriptor Sinks
on page 52.
The RDDM uses PCIe memory read TLPs and Avalon-MM write transactions (which can
be bursting transactions) to move large amounts of data from the host memory in
PCIe space to the local FPGA memory in Avalon-MM space. On the other hand, the
WRDM uses PCIe memory write TLPs and Avalon-MM read transactions to move large
amounts of data from the FPGA memory in Avalon-MM space to the host memory in
PCIe space. The Data Movers' transfers are controlled by descriptors that are provided
to the Data Movers through one of their Avalon-ST sink interfaces. The Data Movers
report the transfers’ status through their Avalon-ST source interfaces.
2.2.4.2. Endpoint Mode
In this mode, the external master (in user logic) sends memory reads and writes
upstream via the Bursting Slave. The following modules are enabled:
• Bursting Slave (in bursting mode)
• Bursting Master (in non-bursting mode)
23
UG-20237 | 2021.02.04
Figure 11. P-Tile Avalon-MM IP in Endpoint Mode

ASTI Avalon-ST Sink M Avalon-MM Master
DMA Application Logic P-Tile Avalon-MM IP
ASTO Avalon-ST Source S Avalon-MM Slave
Qsys Interconnect C Avalon Conduit
S Memory
512
TX
M Custom Bursting ASTO
DMA M S Slave P-Tile
S
Controller (512 bits) Avalon-ST
ASTI Completion Interface PCIe
Re-ordering Hard IP
RX 512
Bursting Master
M DMA CS C
M (non-bursting ASTO
Control/ mode) ASTI
BAR AccessLogic (512 bits)
The external Avalon-MM master can be a custom DMA controller that uses the
Bursting Slave in the IP core to send memory reads and writes upstream. These
memory reads and writes can be up to 512-bytes long. The reordering buffer in the IP
core reorders the Completion TLPs received over the PCIe link and sends them to the
Bursting Slave.
The Bursting Master provides the host with access to the registers and memory in the
Avalon-MM address space of the FPGA. It converts PCIe memory reads and writes to
Avalon-MM reads and writes.
Registers in the custom DMA controller can be programmed by software via the
Bursting Master port.
2.2.4.3. Root Port Mode
In this mode, the IP core needs to be able to process memory read and write TLPs
coming from the DMA controller that resides on the Endpoint side. The following
modules are enabled:
• Bursting Master (in bursting and non-bursting modes)
• Bursting Slave (in non-bursting mode)
• Control Register Access
Figure 12. P-Tile Avalon-MM IP in Root Port Mode
Custom DMA Application Logic P-Tile Avalon-MM IP ASTO Avalon-ST Source M Avalon-MM Master
ASTI Avalon-ST Sink S Avalon-MM Slave
Qsys Interconnect
C Avalon Conduit
Control Register
ASTO
S Access 512
Memory (512 bits) ASTI
SS TX
P-Tile
M Bursting Slave Avalon-ST PCIe
S Local (non-bursting ASTO Completion Interface Hard IP
S Processor M S
mode) ASTI Re-ordering
(512 bits)
512
RX
C C Bursting ASTO
M Bursting S M Master
Slave (512 bits) ASTI
24
UG-20237 | 2021.02.04
The IP core must be able to generate and process configuration reads and writes to
the Endpoint and to the Hard IP configuration registers. This is done via the
Configuration Slave. Since the DMA controller resides on the Endpoint side, its control
registers need to be programmed by the FPGA local processor. Using the Bursting
Slave (in non-bursting mode), the local processor can program the Endpoint control
registers for DMA operations. The Endpoint can also send updates of its DMA status to
the local processor via the Bursting Master.
25
UG-20237 | 2021.02.04
Send Feedback
3. Parameters
This chapter provides a reference for all the parameters that are configurable in the
Intel Quartus Prime IP Parameter Editor for the P-Tile Avalon-MM IP for PCIe.
3.1. Top-Level Settings

Table 12. Top-Level Settings
Parameter Value Default Value Description
Select the lane data rate and lane

width.
The lane data rate and
lane width options shown
here apply to the PCIe
Hard IP interface to the
Gen4x16, Interface - 512- Avalon memory-mapped
bit bridge. For the data rate
Note:
Gen3x16, Interface - 512- and width on the interface
bit Gen4x16, between the Avalon
Hard IP Mode Gen4x8, Interface - 256-bit Interface - 512- memory-mapped bridge
bit and the application logic,
Gen3x8, Interface - 256-bit refer to Table 11 on page
Gen4x4, Interface - 128-bit 15.
Gen3x4, Interface - 128-bit Refer to the P-Tile IP for
PCI Express IP Core
Release Notes for the
Note: matrix of configurations
supported by the P-Tile
Avalon memory-mapped IP
for PCI Express.
Root Port
Port Mode Native Endpoint Specifies the port type.
Native Endpoint
Enable the PHY Reconfiguration

Enable PHY Reconfiguration True/False False
Interface.
Select the frequency of the

Application clock. The options
available vary depending on the
setting of the Hard IP Mode
parameter.
For Gen4 modes, the available
400 MHz clock frequencies are 400 MHz /
350 MHz (for
350 MHz / 250 MHz (for Intel
350 MHz Gen4 mode)
PLD Clock Frequency Agilex) and 350 MHz / 200 MHz
250 MHz 250 MHz (for (for Intel Stratix 10 DX).
125 MHz Gen3 modes)
For Gen3 modes, the available
clock frequencies are 250 MHz /
125 MHz (for Intel Agilex) and
250 MHz / 125 MHz (for Intel
Stratix 10 DX).
For more details, refer to Table 11
on page 15.
continued...
3. Parameters
UG-20237 | 2021.02.04
Enable the Separate Reference

Enable SRIS Mode True/False False Clock with Independent Spread
Spectrum Clocking (SRIS) feature.
Enabling this parameter reduces

the simulation time of Hot Reset
P-Tile Sim Mode True/False False tests by 5 ms.
Do not enable this option if
Note:
you need to run synthesis.
Enable the reset of PCS and

Controller in User Mode for
Endpoint and Bypass Upstream
modes.
When this parameter is True,
depending on the topology, new
signals (p<n>_pld_clrpcs_n)
are exported to the Avalon
Streaming interface.
When this parameter is False
Enable RST of PCS & (default), the IP internally ties off
True/False False
Controller these signals instead of exporting
them.
This feature is only
supported in the X8X8
Note:
Endpoint/Bypass Upstream
topology.
If you have more
questions regarding the
Note: bifurcation feature and its
usage, contact your
Application Engineer.
The following figure shows how to enable Root Port mode:
Figure 13. Intel P-Tile Avalon-MM Top-Level IP Parameter Editor for a Gen3x4 Hard IP
in Root Port Mode
27
3. Parameters
UG-20237 | 2021.02.04
3.2. Core Parameters

Depending on which Hard IP Mode you choose in the Top-Level Settings tab, you
will see different tabs for setting the core parameters.
Figure 14. Intel P-Tile Avalon-MM Top-Level IP Parameter Editor for a x8 Hard IP Mode
If you choose a x8 mode (either Gen4 or Gen3), the PCIe0 Settings and PCIe1 Settings tabs will appear.
3.2.1. Base Address Registers

Table 13. BAR Registers
Parameter Value Description
If you select 64-bit prefetchable memory, 2

contiguous BARs are combined to form a 64-bit
prefetchable BAR; you must set the higher
numbered BAR to Disabled.
Disabled Defining memory as prefetchable allows contiguous
64-bit prefetchable memory data to be fetched ahead. Prefetching memory is
advantageous when the requestor may require
BAR0 Type 64-bit non-prefetchable memory more data from the same region than was
32-bit non-prefetchable memory originally requested. If you specify that a memory
32-bit prefetchable memory is prefetchable, it must have the following 2
attributes:
• Reads do not have side effects such as
changing the value of the data read.
• Write merging is allowed.
Disabled
For a definition of prefetchable memory, refer to
BAR1 Type 32-bit non-prefetchable memory
the BAR0 Type description.
32-bit prefetchable memory
Disabled For a definition of prefetchable memory and a

64-bit prefetchable memory description of what happens when you select the
BAR2 Type
64-bit non-prefetchable memory 64-bit prefetchable memory option, refer to the
32-bit non-prefetchable memory BAR0 Type description.
continued...
28
3. Parameters
UG-20237 | 2021.02.04
Parameter Value Description
Disabled
Disabled
64-bit prefetchable memory For a definition of prefetchable memory and a
description of what happens when you select the
64-bit prefetchable memory option, refer to the
32-bit non-prefetchable memory BAR0 Type description.
Disabled
Specifies the size of the address space accessible

BARn Size 128 Bytes - 16 EBytes to BARn when BARn is enabled.
n = 0, 1, 2, 3, 4 or 5
Disabled
4 KBytes - 12 bits
8 KBytes - 13 bits
16 KBytes - 14 bits
32 KBytes - 15 bits
64 KBytes - 16 bits
128 KBytes - 17 bits Specifies the size of the expansion ROM from 4
Expansion ROM
256 KBytes - 18 bits KBytes to 16 MBytes when enabled.
512 KBytes - 19 bits
1 MByte - 20 bits
2 MBytes - 21 bits
4 MBytes - 22 bits
8 MBytes - 23 bits
16 MBytes - 24 bits
3.2.2. PCI Express and PCI Capabilities Parameters

For each core (PCIe0/PCIe1/PCIe2/PCIe3), the PCI Express / PCI Capabilities tab
contains separate tabs for the device, MSI (Endpoint mode), ACS capabilities (Root
Port mode), slot (Root Port mode), MSI-X, and legacy interrupt pin register
parameters.
29
3. Parameters
UG-20237 | 2021.02.04
Figure 15. PCI Express / PCI Capabilities Parameters
3.2.2.1. Device Capabilities
Table 14. Device Capabilities

Specifies the maximum

payload size supported. This
128 bytes parameter sets the read-
Maximum payload sizes
256 bytes 512 bytes only value of the max
supported
512 bytes payload size supported field
of the Device Capabilities
register.
3.2.2.2. Link Capabilities
Table 15. Link Capabilities

Sets the read-only value of

the port number field in the
Link port number (Root Link Capabilities
0 - 255 1
Port only) register. This parameter is
for Root Ports only. It should
not be changed.
When this parameter is

True, it indicates that the
Endpoint uses the same
physical reference clock that
Slot clock configuration True/False True the system provides on the
connector. When it is False,
the IP core uses an
independent clock regardless
of the presence of a
continued...
30
3. Parameters
UG-20237 | 2021.02.04
reference clock on the

connector. This parameter
sets the Slot Clock
Configuration bit (bit 12) in
the PCI Express Link
Status register.
3.2.2.3. Legacy Interrupt Pin Register
Table 16. Legacy Interrupts Parameters

Enable Legacy Interrupts True/False False Enable Legacy Interrupts

for PF0 (INTx) for PF0 of PCIe0.
Set Interrupt Pin for PF0 NO INT NO INT When Legacy Interrupts are
INTA not enabled, the only option
available is NO INT.
When Legacy Interrupts are
enabled, the only option
available is INTA.
3.2.2.4. MSI Capabilities
Table 17. MSI Capabilities

Enables MSI functionality for

PF0.
If this parameter is True,
PF0 Enable MSI True/False False the Number of MSI
messages requested
parameter will appear
allowing you to set the
number of MSI messages.
Enables or disables MSI

PF0 MSI Extended Data
True/False False extended data capability for
Capable
PF0.
1
Sets the number of
2 messages that the
PF0 Number of MSI 4 application can request in
1 the multiple message
messages requested 8
16 capable field of the Message
Control register.
32
31
3. Parameters
UG-20237 | 2021.02.04
3.2.2.5. MSI-X Capabilities
Table 18. MSI-X Capabilities

Enable MSI-X (Endpoint Enables the MSI-X

True/False False
only) functionality.
System software reads this

field to determine the MSI-X
table size <n>, which is
encoded as <n-1>.
0x0 - 0x7FF (only values of
For example, a returned
MSI-X Table Size powers of two minus 1 are 0
value of 2047 indicates a
valid)
table size of 2048. This field
is read-only.
Address offset:
0x068[26:16]
Points to the base of the

MSI-X table. The lower 3 bits
of the table BAR indicator
(BIR) are set to zero by
MSI-X Table Offset 0x0 - 0xFFFFFFFF 0
software to form a 64-bit
qword-aligned offset. This
field is read-only after being
programmed.
Specifies which one of a

function's BARs, located
beginning at 0x10 in
Configuration Space, is used
Table BAR indicator 0x0 - 0x5 0
to map the MSI-X table into
memory space. This field is
read-only after being
programmed.
Used as an offset from the

address contained in one of
the function's Base Address
registers to point to the base
Pending bit array (PBA) of the MSI-X PBA. The lower
0x0 - 0xFFFFFFFF 0
offset 3 bits of the PBA BIR are set
to zero by software to form
a 32-bit qword-aligned
offset. This field is read-only
after being programmed.
Specifies the function's Base

Address register, located
beginning at 0x10 in
Configuration Space, that
PBA BAR indicator 0x0 - 0x5 0
maps the MSI-X PBA into
memory space. This field is
read-only after being
programmed.
3.2.2.6. Device Serial Number Capability
Table 19. Device Serial Number Capability

Enables the device serial

Enable Device Serial
True/False False number capability. This is an
Number Capability
optional extended capability
32
3. Parameters
UG-20237 | 2021.02.04
that provides a unique

identifier for the PCIe
device.
3.2.3. Device Identification Registers

The following table lists the default values of the Device ID registers. You can use the
parameter editor to change the values of these registers.
Table 20. Device ID Registers

Register Name Range Default Value Description
Sets the read-only value of the

Vendor ID register. This parameter
cannot be set to 0xFFFF per the
Vendor ID 16 bits 0x00001172 PCI Express Base Specification.
Set your own Vendor ID by
Note:
changing this parameter.
Address offset: 0x000.

Device ID register. This register is
Device ID 16 bits 0x00000000 only valid in the Type 0 (Endpoint)
Configuration Space.

Revision ID 8 bits 0x00000001 Revision ID register.

Class Code register.
Class Code 24 bits 0x00FF0000
This parameter cannot be set to
0x0 per the PCI Express Base
Specification.

Subsystem Vendor ID register in
the PCI Type 0 Configuration
Space. This parameter cannot be
Subsystem Vendor ID 16 bits 0x00000000 set to 0xFFFF per the PCI Express
Base Specification. This value is
assigned by PCI-SIG to the device
manufacturer.
Address offset: 0x02C.

Subsystem Device ID register in
Subsystem Device ID 16 bits 0x00000000 the PCI Type 0 Configuration
Space.
Address offset: 0x02C.
33
3. Parameters
UG-20237 | 2021.02.04
3.2.4. Configuration, Debug and Extension Options

Table 21. Configuration, Debug and Extension Options
Specifies the Gen 3

requested phase 2/3 far-end
Gen 3 Requested
TX preset vector. Choosing a
equalization far-end TX 0 - 65535 0x00000004
value different from the
preset vector
default is not recommended
for most designs.
Specifies the Gen 4

requested phase 2/3 far-end
Gen 4 Requested
TX preset vector. Choosing a
equalization far-end TX 0 - 65535 0x00000270
value different from the
preset vector
default is not recommended
for most designs.
If this parameter is True

(default), the refclk1 is
stable after pin_perst and
is free-running. This
parameter must be set to
True for Type A/B/C
systems.
If this parameter is False,
refclk1 is only available
later in User Mode. This
parameter must be set to
Port 1 REFCLK Init Active True/False True
False for Type D systems.
This parameter is only
available in the PCIe1
Settings tab for a X8X8
topology.
If you have more
questions regarding
the bifurcation
Note:
feature and its
usage, contact your
Application Engineer.
Enable the P-Tile Debug

Toolkit for JTAG-based
Enable Debug Toolkit True/False False
System Console debug
access.
Enable HIP dynamic Enable the user Hard IP

reconfiguration of PCIe True/False False reconfiguration Avalon-MM
registers interface.
34
3. Parameters
UG-20237 | 2021.02.04
Figure 16. Configuration, Debug and Extension Parameters (with Debug Toolkit
Enabled)
3.3. Avalon-MM Settings

Table 22. Avalon-MM Parameters
Endpoint Settings
Enable Bursting Slave True/False False Enable bursting Avalon-MM

Mode Slave Interface. This will
enable the Endpoint mode
(where the IP's BAS and
BAM modules are enabled,
but its Data Movers are not
enabled).
Address width of Read {10:64} 64 Address width of Read Data

Data Mover Mover.
Address width of Write {10:64} 64 Address width of Write Data

Data Mover Mover.
Export interrupt conduit True/False False Export internal signals to

interfaces support generation of
Legacy Interrupts/multiple
MSI/MSI-X.
Address width of Bursting {10:64} 64 Only present in Root Port

Master mode.
In Endpoint modes, this
parameter is set by the
largest BAR address width.
Root Port Settings
Avalon-MM address width 32-bit 64-bit Selects the Avalon-MM

64-bit address width.
Address width of 1 - 64 32 Selects the address width of

accessible PCIe memory accessible memory space.
space (TXS)
Enable burst capability True/False False Enable burst capabilities for

for Avalon-MM Master the BAR0 RXM. If this option
Port is set to True, the RXM port
continued...
35
3. Parameters
UG-20237 | 2021.02.04
will be a bursting master.

Otherwise, this RXM will be a
single Dword master.
36
UG-20237 | 2021.02.04
Send Feedback
4. Interfaces
4.1. Overview
The P-Tile Avalon-MM IP for PCIe includes many interface types to implement different
functions.These include:
• High-performance bursting master (BAM) and slave (BAS) Avalon-MM interfaces to
translate between PCIe TLPs and Avalon-MM memory-mapped reads and writes
• Read and Write Data Movers to transfer large blocks of data
• Standard PCIe serial interface to transfer data over the PCIe link
• System interfaces for interrupts, clocking, reset
• Optional reconfiguration interface to dynamically change the value of
Configuration Space registers at run-time
• Optional status interface for debug
Unless otherwise noted, all interfaces to the Application Layer are synchronous to the
rising edge of app_clk. You enable the interfaces using the component IP Parameter
Editor.
Read Data Mover (RDDM) interface: This interface transfers DMA data from the PCIe
system memory to the memory in Avalon-MM address space.
Write Data Mover (WRDM) interface: This interface transfers DMA data from the
memory in Avalon-MM address space to the PCIe system memory.
Bursting Master (BAM) interface: This interface provides host access to the registers
and memory in Avalon-MM address space. The Busting Master module converts PCIe
Memory Reads and Writes to Avalon-MM Reads and Writes.
Bursting Slave (BAS) interface: This interface allows the user application in the FPGA
to access the PCIe system memory. The Bursting Slave module converts Avalon-MM
Reads and Writes to PCIe Memory Reads and Writes.
Control Register Access (CRA) interface: This optional, 32-bit Avalon-MM Slave
interface provides access to the Control and Status registers. You must enable this
interface when you enable address mapping for any of the Avalon-MM slaves or if
interrupts are implemented. The address bus width of this interface is fixed at 15 bits.
The prefix for this interface is cra*.
The modular design of the P-Tile Avalon-MM IP for PCIe lets you enable just the
interfaces required for your application.
4. Interfaces
UG-20237 | 2021.02.04
Table 23. Avalon-MM Interface Summary

Byte Enable Max Outstanding
Avalon-MM Type Data Bus Width Max Burst Size
Granularity Read Request
Bursting Mode: 8
cycles Bursting Mode: 64
Bursting Slave 512 bits dword/byte
Non-Bursting Mode: 1 Non-Bursting Mode: 1
cycle
Bursting Mode: 8
cycles Bursting Mode: 32
Bursting Master 512 bits dword/byte
Non-Bursting Mode: 1 Non-Bursting Mode: 1
cycle
Read Data Mover

512 bits 8 cycles dword N/A
Write Master
Write Data Mover

512 bits 8 cycles dword 32
Read Master
Control Register
32 bits 1 cycle byte 1
Access
Note: The number of read requests issued by the Write Data Mover's Avalon-MM Read
Master is controlled by the assertion of waitrequest by the connected slave(s). The
Read Master can handle 128 outstanding cycles of data. You cannot set this parameter
in Platform Designer. The slave needs to correctly back-pressure the master once it
cannot handle the incoming requests.
Note: The 512-bit Bursting Slave interface does not support transactions where all byte
enables are set to 0.
Note: All transfers of four bytes or more are done in multiples of dwords.
4.2. Clocks and Resets
4.2.1. Interface Clock Signals

Table 24. Interface Clock Signals
Name I/O Description EP/RP Clock Frequency
Native Gen3: 250 MHz

Native Gen4: 350 MHz (Intel
Stratix 10 DX) / 400 MHz
This is the application clock generated (Intel Agilex)
p<n>_app_clk The frequencies given above
from coreclkout_hip or from the
(where n = 0, 1, O EP/RP are the maximum frequencies.
same source as refclk. This is a
2, 3)
per-port signal. The frequencies available vary
depending on the
configurations that the IP is
in. For more details, refer to
Table 11 on page 15.
This is an internal clock only that is

planned to be removed in a future Native Gen3: 250 MHz
release of the Intel FPGA P-tile Avalon Native Gen4: 350 MHz (Intel
coreclkout_hip O EP/RP
Memory-mapped IP for PCI Express. Stratix 10 DX) / 400 MHz
The Application Layer must use the (Intel Agilex)
p<n>_app_clk instead.
continued...
38
4. Interfaces
UG-20237 | 2021.02.04
Name I/O Description EP/RP Clock Frequency
The frequency depends on the data

rate and the number of lanes being
used.
100 MHz ± 300 ppm

When the Enable SRIS Mode
parameter is enabled in the IP
These are the input reference clocks Parameter Editor, the P-Tile
for the IP core. These clocks must be Avalon-MM IP can
free-running. communicate with a link
refclk[1:0] I EP/RP partner whose clock domain is
For more details on how to connect
not synchronized to the
these clocks, refer to the section
refclk domain of the P-Tile.
Clock Sharing in Bifurcation Modes.
In this mode of operation, P-
Tile and its link partner can
both have their own spread
spectrum clocks.
Clock for the hip_reconfig

interface. This is an Avalon-MM
p<n>_hip_reconfig interface. It is an optional interface
that is enabled when the Enable HIP 50 MHz - 125 MHz (range)
_clk (where n = 0, I EP/RP
dynamic reconfiguration of PCIe 100 MHz (recommended)
1, 2, 3) registers option in the PCIe
Configuration, Debug and
Extension Options tab is enabled.
Clock for the PHY reconfiguration

interface. This is an Avalon-MM
interface. This optional interface is
enabled when you turn on the Enable 50 MHz - 125 MHz (range)
xcvr_reconfig_clk I EP/RP
PHY reconfiguration option in the 100 MHz (recommended)
Top-Level Settings tab. This
interface is shared among all the
cores.
4.2.2. Interface Reset Signals

Table 25. Interface Reset Signals
Signal Name Direction Clock EP/RP Description
pin_perst_n Input Asynchrono EP/RP This is an active-low input to the PCIe Hard IP, and
us implements the PERST# function defined by the PCIe
specification.
p<n>_reset_status_n Output Synchronou EP/RP This active-low signal is held low until pin_perst_n has
s been deasserted and the PCIe Hard IP has come out of
reset. This signal is synchronous to p<n>_app_clk.
When port bifurcation is used, there is one such signal for
each interface. The signals are differentiated by the
prefixes p<n>.
p<n>_link_req_rst_n Output Synchronou EP/RP This active-low signal is asserted by the PCIe Hard IP
s when it is about to go into reset.
The Avalon-MM Bridge IP will reset all its PCIe-related
registers and queues including anything related to tags.
It will also stop sending packets to the PCIe Hard IP until
the Bus Master Enable bit is set again. The Bridge will
also ignore any packet received from the PCIe Hard IP.
p<n>_pld_warm_rst_r Input Synchronou EP/RP This active-high signal is asserted by the user logic in
dy s response to p<n>_link_req_rst_n when it has
completed its pre-reset tasks.
continued...
39
4. Interfaces
UG-20237 | 2021.02.04
Signal Name Direction Clock EP/RP Description
Note: When not using this signal, set it to 1'b1.
ninit_done Input Asynchrono EP/RP A "1" on this active-low signal indicates that the FPGA
us device is not yet fully configured. A "0" indicates the
device has been configured and is in normal operating
mode.
Intel recommends using the output of the Reset Release
Intel Fpga IP to drive this ninit_done input. For more
details on this IP, refer to the Application Note AN891 at
https://www.intel.com/content/dam/www/
programmable/us/en/pdfs/literature/an/an891.pdf
4.3. Avalon-MM Interface

The figures below provide the top-level block diagrams of the P-Tile Avalon-MM IP with
all interfaces while operating in Endpoint mode with Data Movers or in Endpoint mode.
These interfaces are described in more details in following sections.
40
4. Interfaces
UG-20237 | 2021.02.04
Figure 17. P-Tile Avalon-MM IP for PCIe in Endpoint Mode with Data Movers Top-Level
Block Diagram
P-Tile Avalon-MM IP for PCI Express

p0_rddm_write_o refclk[1:0]
p0_rddm_address_o[63:0] coreclkout_hip Clocks
p0_rddm_write_data_o[511:0] p0_app_clk
Read Data Mover Interface: p0_rddm_burst_count_o[3:0]
Writes Data from Host p0_rddm_byte_enable_o[63:0]
Memory to FPGA Memory p0_rddm_wait_request_i
p0_rddm_pfnum_o[1:0]
p0_rddm_tx_valid_o
p0_rddm_tx_data_o[31:0]
p0_hip_reconfig_clk
Read Data Mover p0_rddm_prio_ready_o p0_hip_reconfig_address[20:0]
p0_rddm_prio_valid_i p0_hip_reconfig_read Hard IP
Priority Descriptor p0_hip_reconfig_readdata[7:0]
Queue p0_rddm_prio_data_i[173:0] Reconfiguration
p0_hip_reconfig_readdatavalid (Optional)
p0_hip_reconfig_write
Read Data Mover p0_rddm_desc_ready_o p0_hip_reconfig_writedata[7:0]
Normal Descriptor p0_rddm_desc_valid_i p0_hip_reconfig_waitrequest
Queue p0_rddm_desc_data_i[173:0]
tx_p_out[15:0]
p0_wrdm_read_o tx_n_out[15:0]
p0_wrdm_response_i[1:0] Serial Data
p0_wrdm_byteenable_o[63:0] rx_p_in[15:0]
p0_wrdm_address_o[63:0] rx_n_in[15:0]
Write Data Mover Interface: p0_wrdm_readdata_i[511:0]
Writes Data from FPGA p0_wrdm_burst_count_o[3:0] xcvr_reconfig_clk
Memory to Host Memory p0_wrdm_wait_request_i
p0_wrdm_readdatavalid_i xcvr_reconfig_address[25:0]
p0_wrdm_pfnum_o[1:0] xcvr_reconfig_read
xcvr_reconfig_readdata[7:0] PHY
p0_wrdm_tx_valid_o Reconfiguration
p0_wrdm_tx_data_o[31:0] xcvr_reconfig_readdatavalid (Optional)
xcvr_reconfig_write
Write Data Mover p0_wrdm_prio_ready_o
xcvr_reconfig_writedata[7:0]
Priority Descriptor p0_wrdm_prio_valid_i xcvr_reconfig_waitrequest
Queue p0_wrdm_prio_data_i[173:0]
Write Data Mover p0_wrdm_desc_ready_o

Normal Descriptor p0_wrdm_desc_valid_i
Queue p0_wrdm_desc_data_i[173:0]
p0_bus_master_enable_o[1:0]
p0_tl_cfg_func_o[2:0]
p0_bam_bar_o[2:0] Configuration Output
p0_tl_cfg_add_o[4:0] Interface
p0_bam_pfnum_o[1:0]
p0_tl_cfg_ctl_o[15:0]
p0_bam_waitrequest_i
p0_bam_response_i[1:0]
Bursting Avalon-MM p0_bam_address_o[BAM_ADDR_WIDTH-1:0]
Master Interface p0_bam_byteenable_o[63:0]
p0_bam_read_o
p0_bam_readdata_i[511:0]
p0_bam_readdatavalid_i
p0_bam_write_o
p0_bam_writedata_o[511:0]
p0_bam_burstcount_o[3:0]
p0_pld_link_req_rst_o
p0_pld_warm_rst_rdy_i
p0_reset_status_n
Reset pin_perst_n
ninit_done
dummy_user_avmm_rst
41
4. Interfaces
UG-20237 | 2021.02.04
Figure 18. P-Tile Avalon-MM IP for PCIe in Endpoint Mode Top-Level Block Diagram
P-Tile Avalon-MM IP for PCI Express

refclk[1:0]
coreclkout_hip Clocks
p0_app_clk
p0_tl_cfg_func_o[2:0]
Configuration Output p0_tl_cfg_add_o[4:0]
Interface
p0_tl_cfg_ctl_o[15:0]
p0_hip_reconfig_clk
p0_hip_reconfig_address[20:0]
p0_hip_reconfig_read Hard IP
p0_hip_reconfig_readdata[7:0] Reconfiguration
p0_hip_reconfig_readdatavalid (Optional)
p0_hip_reconfig_write
p0_hip_reconfig_writedata[7:0]
p0_hip_reconfig_waitrequest
tx_p_out[15:0]
p0_bas_waitrequest_o tx_n_out[15:0]
p0_bas_pfnum_i[1:0] Serial Data
p0_bas_byteenable_i[63:0] rx_p_in[15:0]
p0_bas_address_i[63:0] rx_n_in[15:0]
Bursting Avalon-MM p0_bas_readdata_o[511:0]
Slave Interface p0_bas_read_i xcvr_reconfig_clk
p0_bas_readdatavalid_o
p0_bas_write_i xcvr_reconfig_address[25:0]
xcvr_reconfig_read
p0_bas_writedata_i[511:0] PHY
p0_bas_burstcount_i[3:0] xcvr_reconfig_readdata[7:0]
Reconfiguration
p0_bas_response_o[1:0] xcvr_reconfig_readdatavalid (Optional)
xcvr_reconfig_write
xcvr_reconfig_writedata[7:0]
xcvr_reconfig_waitrequest
p0_bus_master_enable_o[1:0]
p0_bam_bar_o[2:0]
p0_bam_pfnum_o[1:0]
p0_bam_waitrequest_i
p0_bam_response_i[1:0]
Bursting Avalon-MM p0_bam_address_o[BAM_ADDR_WIDTH-1:0]
Master Interface p0_bam_byteenable_o[63:0]
p0_bam_read_o
p0_bam_readdata_i[511:0]
p0_bam_readdatavalid_i
p0_bam_write_o
p0_bam_writedata_o[511:0]
p0_bam_burstcount_o[3:0]
p0_pld_link_req_rst_o
p0_pld_warm_rst_rdy_i
p0_reset_status_n
Reset pin_perst_n
ninit_done
dummy_user_avmm_rst
4.3.1. Endpoint Mode Interface (512-bit Avalon-MM Interface)

Table 26. Avalon-MM Interface Summary
Avalon-MM Type Data Bus Width Max Burst Size Byte Enable Max Outstanding
Bursting Slave 512 bits 8 cycles byte 64
Bursting Master 512 bits 8 cycles byte 32

continued...
42
4. Interfaces
UG-20237 | 2021.02.04
Avalon-MM Type Data Bus Width Max Burst Size Byte Enable Max Outstanding
Read Data Mover 512 bits 8 cycles dword N/A

Write Master
Write Data Mover 512 bits 8 cycles dword 128

Read Master
Control Register 32 bits 1 cycle byte 1

Access
These interfaces are standard Avalon interfaces. For timing diagrams, refer to the
Avalon Interface Specifications.
Note: The number of read requests issued by the Write Data Mover's Avalon-MM Read
Master is controlled by the assertion of waitrequest by the connected slave(s). The
Read Master can handle 128 outstanding cycles of data. You cannot set this parameter
in Platform Designer. The slave needs to correctly back-pressure the master once it
cannot handle the incoming requests.
Note: The 512-bit Bursting Slave interface does not support transactions where byte enables
are set to 0.
Related Information
Avalon Interface Specifications
4.3.1.1. Bursting Avalon-MM Master and Conduit
The Bursting Avalon-MM Master module has one user-visible Avalon-MM Master
interface.
You enable this interface by turning On the Enable Bursting Avalon-MM Master
interface option in the Avalon-MM Settings tab of the IP Parameter Editor.
Table 27. Bursting Avalon-MM Master and Conduit

Signal Name Direction Description Platform Designer
Interface Name
Physical function number

• PF0: bam_pfnum_o[1:0] = 2'b00
bam_pfnum_o[1:0] O
• Others: bam_pfnum_o[1:0] =
Reserved
This bus contains the BAR address for a

particular TLP. This bus acts as an
extension of the standard address bus.
000: Memory BAR 0
001: Memory BAR 1
bam_bar_o[2:0] O 010: Memory BAR 2
011: Memory BAR 3
100: Memory BAR 4
101: Memory BAR 5
110: Reserved
111: Expansion ROM BAR
When asserted, indicates that the Avalon-

bam_waitrequest_i I MM slave is not ready to respond to a bam_master
request.
continued...
43
4. Interfaces
UG-20237 | 2021.02.04

Interface Name
waitrequestAllowance = 8
The master can still issue 8 transfers after
bam_waitrequest_i is asserted.
The width of the Bursting Master’s

address bus is the maximum of the
bam_address_o[BAM_ADDR_W widths of all the enabled BARs.
O
IDTH-1:0] For BARs narrower than the widest BAR,
the address bus’ additional most
significant (MSB) bits are driven to 0.
Specify the valid bytes of

bam_writedata_o[511:0]. Each bit
corresponds to a byte in
bam_writedata_o[511:0].
For single-cycle read bursts and for all
write bursts, all contiguous sets of
bam_byteenable_o[63:0] O
enabled bytes are supported.
For multi-cycle read bursts, all bits of
bam_byteenable_o[63:0] are
asserted, regardless of the First Byte
Enable (BE) and Last BE fields of the
corresponding TLP.
When asserted, indicates the master is

bam_read_o O
requesting a read transaction.
bam_readdata_i[511:0] I Read data bus
Asserted by the slave to indicate that the

bam_readdata_i[511:0] bus contains
bam_readdatavalid_i I
valid data in response to a previous read
request.
When asserted, indicates the master is

bam_write_o O
requesting a write transaction.
bam_writedata_o[511:0] O Data signals for write transfers.
The master uses these signals to indicate

bam_burstcount_o[3:0] O
the number of transfers in each burst.
00 : OKAY - successful response for a

transaction.
bam_response_i[1:0] I 01 : RESERVED
10 : SLAVEERROR
11 : DECODEERROR
4.3.1.1.1. Bursting Avalon-MM Master and Conduit in Non-Bursting Mode
In non-bursting mode, the Bursting Avalon-MM Master module has the same interface
as in bursting mode, except for some limitations in the size of transactions as
described below:
44
4. Interfaces
UG-20237 | 2021.02.04
• Burst count must be 1.

• The request size ranges from 1 to 16 dwords with the following limitations:
— The address and size combination must generate a TLP that fits in one 512-bit
chunk of data. For example, if the address starts at dword 15 of a 512-bit
transaction, only one dword of data transfer is allowed. If the address starts at
dword 0, all data transfer sizes up to 16 dwords are possible. The same rule
applies to read completions.
— Byte enables are supported for a transfer size of one dword. For larger
transfer sizes, dword enables apply.
• If non-bursting mode is enabled, sending a TLP larger than 64 bytes targeting this
interface causes the interface to misbehave. In this case, a reset is required to
allow the interface to recover.
• One outstanding read at a time. Incoming RX read/write TLPs will be delayed while
a downstream outstanding read exists.
4.3.1.2. Bursting Avalon-MM Slave and Conduit
The Bursting Avalon-MM Slave module has one user-visible Avalon-MM slave interface.
You enable this interface by turning On the Enable Bursting Avalon-MM Slave
interface option in the IP Parameter Editor.
For more details on these interface signals, refer to the Avalon Interface
Specifications.
Table 28. Bursting Avalon-MM Slave and Conduit

Interface Name
Physical function number

• PF0:
bas_pfnum_i[1:0] =
bas_pfnum_i[1:0] I 2'b00
• Others:
bas_pfnum_i[1:0] =
Reserved
When asserted, indicates

that the Avalon-MM slave is
not ready to respond to a
request.
bas_waitrequest_o O waitrequestAllowance = 0
The master cannot issue any
transfer after
bas_waitrequest_o is
asserted.
bas_master
Specify the byte address
bas_address_i[63:0] I regardless of the data width
of the master.

bas_writedata_i[511:0]
. Each bit corresponds to a
bas_byteenable_i[63:0] I
byte in
bas_writedata_i[511:0]
.
continued...
45
4. Interfaces
UG-20237 | 2021.02.04

Interface Name
For single-cycle read bursts

and for all write bursts, all
contiguous sets of enabled
bytes are supported.
For burst read transactions,
the
bas_byteenable_i[63:0]
must be
64'hFFFF_FFFF_FFFF_FFFF.
When asserted, indicates the

bas_read_i I master is requesting a read
transaction.
Ensure that disabled bytes

bas_readdata_o[511:0] O
do not contain stale data.
maximumPendingReadTrans
actions: 64
bas_readdatavalid_o O The maximum number of
pending reads that the
Avalon-MM slave can queue
up is 64.
These bits contain the

response status for any
transaction happening on
the BAS interface:
• 00: OKAY - Successful
response for a
transaction.
• 01: RESERVED - This
bas_response_o[1:0] O encoding is reserved.
• 10: SLAVEERROR - Error
from an endpoint slave.
Indicates an unsuccessful
transaction.
• 11: DECODEERROR -
Indicates an attempted
access to an undefined
location.

bas_write_i I master is requesting a write
transaction.
Data signals for write

bas_writedata_i[511:0] I
transfers.
The master uses these

signals to indicate the
bas_burstcount_i[3:0] I
number of transfers in each
burst.
4.3.1.2.1. Bursting Avalon-MM Slave and Conduit in Non-Bursting Mode
The Bursting Avalon-MM Slave module supports bursting mode while operating in Root
Port mode.
This module can also operate in non-bursting mode. In this mode, the module
interface is the same as in bursting mode except that it has limitations in the size of
transactions as described below:
46
4. Interfaces
UG-20237 | 2021.02.04
• Burst count must be 1.

• The request size ranges from 1 to 16 dwords with the following limitations:
— The address and size combination must generate a TLP that fits in one 512B
chunk of data. For example, if the address starts at dword 15 of a 512B
transaction, only one dword of data transfer is allowed. If the address starts at
dword 0, all data transfer sizes up to 16 dwords are possible. The same rule
applies to read completions.
— Byte enables are supported for a transfer size of one dword. For larger
transfer sizes, dword enables apply.
• One outstanding read at a time (back-pressures the Avalon-MM Master while the
outstanding read exists).
4.3.1.3. Read Data Mover
The Read Data Mover has four user-visible interfaces:

• One Avalon-MM Write Master with sideband signals to write data to the Avalon
domain.
• Two Avalon-ST Sinks to receive descriptors. One acts as a queue for priority
descriptors, and the other acts as a queue for normal descriptors.
• One Avalon-ST Source to report status.
4.3.1.3.1. Read Data Mover Avalon-MM Write Master and Conduit
This interface provides the Read data from the Host memory to the user application.
The rddm_address_o value is set within the descriptor destination address.
Table 29. Read Data Mover Avalon-MM Write Master and Conduit
Interface Name
Physical function number.

• PF0:
rddm_pfnum_o[1:0] =
rddm_pfnum_o[1:0] O 2'b00 rddm_conduit
• Others:
rddm_pfnum_o[1:0] =
Reserved

request.
rddm_waitrequest_i I waitrequestAllowance = 16
The master can still issue 16
transfers after
rddm_waitrequest_i is
asserted. rddm_master

rddm_write_o O master is requesting a write
transaction.

rddm_address_o[63:0] O regardless of the data width
of the master.
continued...
47
4. Interfaces
UG-20237 | 2021.02.04

Interface Name

rddm_burstcount_o[3:0] O
burst.

rddm_writedata_o[511:0
rddm_byteenable_o[63:0 ]. Each bit corresponds to a
O
] byte in
rddm_writedata_o[511:0
].
rddm_writedata_o[511:0 Data signals for write

O
] transfers.
4.3.1.3.2. Read Data Mover Avalon-ST Descriptor Sinks
The Read Data Mover has two Avalon-ST sink interfaces to receive the descriptors that
define the data transfers to be executed. One of the interfaces receives descriptors for
normal data transfers, while the other receives descriptors for high-priority data
transfers.
The descriptor format for the Read Data Mover is described in the section Descriptor
Format for Data Movers.
Note: The user application is responsible for performing the scheduling between priority and
normal queues. No arbitration is performed inside the Read Data Mover.
Table 30. Read Data Mover Avalon-ST Normal Descriptor Sink Interface
Interface Name
rddm_desc_ready_o O When asserted, this ready

signal indicates the normal
descriptor queue in the Read
Data Mover is ready to
accept data. The ready
latency of this interface is 3
cycles.
rddm_desc_valid_i I When asserted, this signal

qualifies valid data on any
cycle where data is being
transferred to the normal rddm_desc
descriptor queue. On each
cycle where this signal is
active, the queue samples
the data.
rddm_desc_data_i[173:0 I [173:160]: reserved. Should

] be tied to 0.
[159:152]: descriptor ID
[151:149] : application
specific
[148] : single destination (3)
continued...
(3) When the single destination bit is set, the same destination address is used for all the
transfers. If the bit is not set, the address increments for each transfer.
48
4. Interfaces
UG-20237 | 2021.02.04

Interface Name
[147] : reserved
[146] : reserved
[145:128]: number of
dwords to transfer up to 1
MB
[127:64]: destination
Avalon-MM address
[63:0]: source PCIe address
Table 31. Read Data Mover Avalon-ST Priority Descriptor Sink Interface
Interface Name
rddm_prio_ready_o O When asserted, this ready

signal indicates the priority
descriptor queue in the Read
Data Mover is ready to
cycles.
rddm_prio_valid_i I When asserted, this signal

transferred to the priority
the data.
rddm_prio
rddm_prio_data_i[173:0 I [173:160]: reserved. Should
] be tied to 0.
specific
[148] : single destination
[147] : reserved
[146] : reserved
MB
[127:64]: destination
Avalon-MM address
[63:0]: source PCIe address
The Read Data Mover internally supports two queues of descriptors. The priority queue
has absolute priority over the normal queue. Use it carefully to avoid starving the
normal queue.
If the Read Data Mover receives a descriptor on the priority interface while processing
a descriptor from the normal queue, it switches to processing descriptors from the
priority queue as soon as it has completed the current descriptor. The Read Data
Mover resumes processing the descriptors from the normal queue once the priority
queue is empty. Do not use the same descriptor ID simultaneously in the two queues
as there would be no way to distinguish them on the Status Avalon-ST source
interface.
49
4. Interfaces
UG-20237 | 2021.02.04
The Read Data Mover handles one descriptor at a time. When a descriptor has been
processed (the memory command has been issued to the PCIe link), the Read Data
Mover will read the next descriptor from the priority or normal descriptor interface.
Note: There is no buffer to store descriptors inside the Read Data Mover. In Intel's DMA
design example, the buffer is located in the external DMA controller and supports up
to 128 descriptors.
Software should only send new descriptors when the Read Data Mover has processed
all previously sent descriptors. The P-Tile Avalon-MM IP indicates the completion of the
Read Data Mover's data processing by performing an immediate write to the system
memory using its Write Data Mover. For more details, refer to the Read DMA Example
section in the P-tile Avalon Memory Mapped (Avalon-MM) IP for PCI Express Design
Example User Guide (see the link in the Related Information below).
Related Information
Read DMA Example
4.3.1.3.3. Read Data Mover Status Avalon-ST Source
Table 32. Read Data Mover Status -ST Source

Interface Name
rddm_tx_data_o[31:0] O [31:16]: reserved

[15]: error
[14:12]: application specific
[11:9] : reserved rddm_tx
[8] : priority
rddm_tx_valid_o O Valid status signal
This interface does not have a ready input. The application logic must always be ready
to receive status information for any descriptor that it has sent to the Read Data
Mover.
The Read Data Mover copies over the application specific bits in the rddm_tx_data_o
bus from the corresponding descriptor. A set priority bit indicates that the descriptor is
from the priority descriptor sink.
A status word is output on this interface when the processing of a descriptor has
completed, including the reception of all completions for all memory read requests.
4.3.1.4. Write Data Mover
The Write Data Mover has four user visible interfaces:

• One Avalon-MM Read Master with sideband signals to read data from the Avalon
domain.
• Two Avalon-ST Sinks to receive descriptors. One Sink acts as a queue for priority
descriptors, and the other acts as a queue for normal descriptors.
• One Avalon-ST Source to report status
50
4. Interfaces
UG-20237 | 2021.02.04
4.3.1.4.1. Write Data Mover Avalon-MM Read Master and Conduit
This interface reads data from the Avalon-MM Read Master interface and writes it to
the Host memory.
The wrdm_address_o value is set within the descriptor source address.
Table 33. Write Data Mover Avalon-MM Read Master and Conduit
Interface Name
Physical function number.

• PF0:
wrdm_pfnum_o[1:0] =
wrdm_pfnum_o[1:0] O 2'b00
• Others:
wrdm_pfnum_o[1:0] =
Reserved

request.
wrdm_waitrequest_i I waitrequestAllowance = 4
The master can still issue 4
transfers after
wrdm_waitrequest_i is
asserted.

wrdm_read_o O master is requesting a read
transaction.

wrdm_address_o[63:0] O regardless of the data width
of the master.

wrdm_burstcount_o[3:0] O
burst. wrdm_master

wrdm_writedata_o[511:0
wrdm_byteenable_o[63:0 ]. Each bit corresponds to a
O
] byte in
wrdm_writedata_o[511:0
].
Asserted by the slave to

indicate that the
wrdm_readdata_i[511:0]
wrdm_readdatavalid_i I
signals contain valid data in
response to a previous read
request.
Data signals for read

wrdm_readdata_i[511:0] I
transfers.
The response signals are

wrdm_response_i[1:0] I optional signals that carry
the response status.
continued...
51
4. Interfaces
UG-20237 | 2021.02.04

Interface Name
Because the signals

are shared, an
interface cannot
Note: issue or accept a
write response and a
read response in the
same clock cycle.
The following encodings are
available:
• 00: OKAY - Successful
response for a
transaction.
• 01: RESERVED -
Encoding is reserved.
• 10: SLAVEERROR - Error
from an endpoint slave.
Indicates an unsuccessful
transaction.
• 11: DECODEERROR -
Indicates an attempted
access to an undefined
location.
For read responses:
• One response is sent
with each readdata. A
read burst length of N
results in N responses. It
is not valid to produce
fewer responses, even in
the event of an error. It
is valid for the response
signal values to be
different for each
readdata in the burst.
• The interface must have
read control signals.
Pipeline support is
possible with the
readdatavalid signal.
• On a read error, the
corresponding readdata
is a "don't care".
4.3.1.4.2. Write Data Mover Avalon-ST Descriptor Sinks
The Write Data Mover has two Avalon-ST sink interfaces to receive the descriptors that
define the data transfers to be executed. One of the interfaces receives descriptors for
normal data transfers, while the other receives descriptors for high-priority data
transfers.
The descriptor format for the Write Data Mover is described in the section Descriptor
Formats for Data Movers.
Note: The user application is responsible for performing the scheduling between priority and
normal queues. No arbitration is performed inside the Write Data Mover.
52
4. Interfaces
UG-20237 | 2021.02.04
Table 34. Write Data Mover Avalon-ST Normal Descriptor Sink Interface
Interface Name
wrdm_desc_ready_o O When asserted, this ready

signal indicates the normal
descriptor queue in the
Write Data Mover is ready to
cycles.
wrdm_desc_valid_i I When asserted, this signal

transferred to the normal
the data.
I [173:160]: reserved. Should

wrdm_desc
wrdm_desc_data_i[173:0
] be tied to 0.
specific
[148] : reserved
[147] : single source (4)
[146] : immediate (5)
MB
[127:64]: destination PCIe
address
[63:0]: source Avalon-MM
address / immediate data
Table 35. Write Data Mover Avalon-ST Priority Descriptor Sink Interface
Interface Name
wrdm_prio_ready_o O When asserted, this ready

signal indicates the priority
descriptor queue in the
Write Data Mover is ready to
wrdm_prio
cycles.
wrdm_prio_valid_i I When asserted, this signal

transferred to the priority
continued...
(4) When the single source bit is set, the same source address is used for all the transfers. If the
bit is not set, the address increments for each transfer. Note that in single source mode, the
PCIe address and Avalon-MM address must be 64-byte aligned.
(5) When set, the immediate bit indicates immediate writes. Immediate writes of one or two
dwords are supported. For immediate transfers, bits [31:0] or [63:0] contain the payload for
one- or two-dword transfers respectively. The two-dword immediate writes cannot cross a 4k
boundary.
53
4. Interfaces
UG-20237 | 2021.02.04

Interface Name

the data.
wrdm_prio_data_i[173:0 I [173:160]: reserved. Should

] be tied to 0.
specific
[148] : reserved
[147] : single source
[146] : immediate
MB
[127:64]: destination PCIe
address
[63:0]: source Avalon-MM
address / immediate data
The Write Data Mover internally supports two queues of descriptors. The priority
queue has absolute priority over the normal queue, so it should be used carefully to
avoid starving the normal queue.
If the Write Data Mover receives a descriptor on the priority interface while processing
a descriptor from the normal queue, it switches to processing descriptors from the
priority queue after it has completed processing the current descriptor. The Write Data
Mover resumes processing descriptors from the normal queue once the priority queue
is empty. Do not use the same descriptor ID simultaneously in the two queues as
there would be no way to distinguish them on the Status Avalon-ST source interface.
The Write Data Mover handles one descriptor at a time. When a descriptor has been
processed, the Write Data Mover will read the next descriptor from the priority or
normal descriptor interface.
Note: There is no buffer to store descriptors inside the Write Data Mover. In Intel's DMA
design example, the buffer is located in the external DMA controller and supports up
to 128 descriptors.
Software should only send new descriptors when the Write Data Mover has processed
all previously sent descriptors. The Write Data Mover indicates the completion of the
its data processing by performing an immediate write to the system memory using the
last descriptor in the descriptor table. For more details, refer to the Write DMA
Example section in the P-tile Avalon Memory Mapped (Avalon-MM) IP for PCI Express
Design Example User Guide (see the link in the Related Information below).
Related Information
Write DMA Example
54
4. Interfaces
UG-20237 | 2021.02.04
4.3.1.4.3. Write Data Mover Status Avalon-ST Source
Table 36. Write Data Mover Status Avalon-ST Source

Interface Name
wrdm_tx_data_o[31:0] O [31:16]: reserved

[15]: error
[14:12] : application specific
[11:9] : reserved wrdm_tx
[8] : priority bit
wrdm_tx_valid_o O Valid status signal
This interface does not have a ready input. The application logic must always be ready
to receive status information for any descriptor that it has sent to the Write Data
Mover.
The ready latency does not matter because there is no ready input.
The Write Data Mover copies over the application specific bits in the wrdm_tx_data_o
bus from the corresponding descriptor. A set priority bit indicates that the descriptor
was from the priority descriptor sink.
4.3.1.5. Descriptor Format for Data Movers
The Read and Write Data Movers uses descriptors to transfer data. The descriptor
format is fixed and specified below:
Table 37. Descriptor Format for Data Movers

Signals Description (for Read Data Mover Write Data Mover
rddm_desc_data_i or
wrdm_desc_data_i)
[173:160]: reserved N/A N/A
[159:152]: descriptor ID ID of the descriptor ID of the descriptor
[151:149]: application-specific Application-specific bits. Application-specific bits.

Example of an Intel application is Example of an Intel application is
provided below. provided below.
[148]: single destination When the single destination bit is set, N/A
the same destination address is used
for all the transfers. If the bit is not
set, the address increments for each
transfer.
[147]: single source N/A When the single source bit is set, the
same source address is used for all the
transfers. If the bit is not set, the
address increments for each transfer.
Note that in single source mode, the
PCIe address and Avalon-MM address
must be 64-byte aligned.
[146]: immediate N/A When set, the immediate bit indicates

immediate writes. Immediate writes of
one or two dwords are supported.
continued...
55
4. Interfaces
UG-20237 | 2021.02.04
Signals Description (for Read Data Mover Write Data Mover

rddm_desc_data_i or
wrdm_desc_data_i)
For immediate transfers, bits [31:0] or

[63:0] contain the payload for one- or
two-dword transfers respectively. The
two-dword immediate writes cannot
cross a 4k boundary.
This can be used for MSI/MSI-X for
example.
[145:128]: transfer size Number of dwords to transfer (up to 1 Number of dwords to transfer (up to 1
MB). MB).
[127:64]: destination address Avalon-MM address PCIe Address
[63:0]: source address PCIe Address Avalon-MM address
Application-Specific Bits
Three application-specific bits (bits [151:149] ) from the Write Data Mover and Read
Data Mover Status Avalon-ST Source interfaces control when interrupts are generated.
Table 38. Encodings for Application-Specific Bits

Bit [151] Bit [150] Bit [149] Action
0 1 1 Interrupt always
0 1 0 Interrupt if error
0 0 1 No interrupt
0 0 0 No interrupt and drop status

word
The External DMA Controller makes the decision whether to drop the status word and
whether to generate an interrupt as soon as it receives the status word from the Data
Mover. When the generation of an interrupt is requested, and the corresponding RI or
WI register does enable interrupts, the DMA Controller generates the interrupt. It does
so by queuing an immediate write to the Write Data Mover's descriptor queue
(specified in the corresponding interrupt control register) using the MSI address and
message data provided in that register.
4.3.1.6. Avalon-MM DMA Operations
Avalon-MM DMA operations are used to transfer large blocks of data. The P-Tile
Avalon-MM IP for PCIe can support DMA operations with an external descriptor
controller implemented in the user application.
To interface to the DMA logic included in the P-Tile Avalon-MM IP for PCIe, the custom
DMA descriptor controller must implement the following functions:
• It must provide the descriptors to the Read Data Mover and Write Data Mover in
the P-Tile IP.
• It must process the status that the DMA Avalon-MM Read and Write masters
provide.
The following figure shows the Avalon-MM DMA Bridge when a custom external
descriptor controller drives the Read and Write Data Movers.
56
4. Interfaces
UG-20237 | 2021.02.04
Figure 19. Avalon-MM DMA Bridge Block Diagram with Externally Instantiated
Descriptor Controller
Intel FPGA
Avalon-MM DMA with
external Descriptor P-Tile Avalon-MM IP for PCI Express IP Core
Controller
Avalon-MM DMA Bridge (Soft Logic)
Avalon-MM (rddm_master) Read Data Mover

512 Bits Write Master
Avalon-ST
(rddm_desc, rddm_prio) Read Data Mover Read
Descriptor Sinks Data Mover MRd, CpID
174 Bits
PCIe
Hard IP
Avalon-ST (rddm_tx) Read Data Mover
Custom 32 Bits Status Source
Descriptor Controller
(Implemented in PCIe X16
Avalon-ST TX
FPGA Fabric) (wrdm_desc, wrdm_prio) Write Data Mover
174 Bits Descriptor Sinks
Hard IP Interface
Avalon-ST (wrdm_tx) Write Data Mover Write MWr
32 Bits Status Source Data Mover
Avalon-MM (wrdm_master) Write Data Mover

512 Bits Read Master
PCIe
RX X16
Avalon-MM (bam_*) 512 Bits

MWr, Mrd, CpID
Non-Bursting Avalon-MM Masters
This configuration includes the PCIe Read DMA and Write DMA Data Movers. The
custom DMA descriptor controller must connect to the following Data Mover interfaces:
• PCIe Read Descriptor Sinks: These are two 174-bit, Avalon-ST sink interfaces (for
normal and priority descriptors). The custom DMA descriptor controller drives read
descriptor table entries on this bus. For more details on this interface, refer to
Read Data Mover Avalon-ST Descriptor Sinks on page 48.
• PCIe Write Descriptor Sinks: These are two 174-bit, Avalon-ST sink interfaces (for
normal and priority descriptors). The custom DMA descriptor controller drives
write descriptor table entries on this bus. For more details on this interface, refer
to Write Data Mover Avalon-ST Descriptor Sinks on page 52.
• PCIe Read Data Mover Status Source: The Read Data Mover reports status to the
custom DMA descriptor controller on this interface. For more details on this
interface, refer to Read Data Mover Status Avalon-ST Source on page 50.
• PCIe Write Data Mover Status Source: The Write Data Mover reports status to the
custom DMA descriptor controller on this interface. For more details on this
interface, refer to Write Data Mover Status Avalon-ST Source on page 55.
57
4. Interfaces
UG-20237 | 2021.02.04
4.3.2. Root Port Mode Interface (256-bit Avalon-MM Interface)

In Gen3 x4 and Gen4 x4 Root Port modes, the IP core uses the 256-bit Avalon-MM
bridge instead of the 512-bit Avalon-MM bridge for the performance purpose. In Root
Port mode, DMA functionalities are not available. The table below shows the interfaces
for the P-tile 256-bit Avalon-MM bridge.
Table 39. Summary of Interfaces for the 256-bit Avalon-MM Bridge

Interface Name Data Width Burst Count Width Byte Enable Width Wait Request
Bursting Master 256 5 32 Yes
Non-Bursting Slave 32 N/A 4 Yes

(optional)
Bursting Slave 256 5 32 Yes

(optional)
Control Register 32 N/A 4 Yes

Access (CRA)
4.3.2.1. High Performance Avalon-MM Slave (HPTXS) Interface
The High Performance Avalon-MM Slave has a 256-bit-wide data bus. It supports up to
16-cycle bursts with dword granularity byte enable on the first and last cycles of a
write burst and for single-cycle read bursts. It also supports optional address mapping
when the address bus is less than 64-bit wide.
This interface is optional. You enable it by turning On the Enable Bursting Slave
option in the GUI.
Table 40. High Performance Avalon-MM Slave (HPTXS) Interface

Interface Name
hptxs_address_i I Byte address. Bits [4:0] are

[hptxs_address_width_h assumed to be zeros.
wtcl-1:0]
hptxs_byteenable_i I Specifies the valid bytes for

[31:0] a write command.
hptxs_read_i I When asserted, specifies a

TX Avalon-MM slave read
request.
hptxs_slave
hptxs_readdata_o[255:0 O This bus contains the read
] completion data.
hptxs_write_i I When asserted, specifies a

TX Avalon-MM slave write
request.
hptxs_writedata_i[255: I This bus contains the

0] Avalon-MM data for a write
command.
continued...
58
4. Interfaces
UG-20237 | 2021.02.04

Interface Name
hptxs_waitrequest_o O When asserted, indicates

that the Avalon-MM slave
port is not ready to respond
to a read or write request.
hptxs_readdatavalid_o O When asserted, indicates

that the read data is valid.
hptxs_burstcount_i[4:0 I When asserted, the value on

] the response signal is a valid
write response.
Writeresponsevalid is
only asserted one clock cycle
or more after the write
command is accepted.
There is at least a one clock
cycle latency from command
acceptance to the assertion
of writeresponsevalid.
4.3.2.2. High Performance Avalon-MM Master (HPRXM) Interface
The bursting Avalon-MM master is always enabled in Root Port mode and is not
associated with any BAR. Packets targeting addresses outside of the range of the base
and limit registers are forwarded to the host via the HPRXM master. The bursting
Avalon-MM master has a 256-bit-wide data bus and supports up to 16-cycle bursts
with dword granularity byte enable on the first and last cycles of a write burst and on
single-cycle read bursts. Byte granularity access is supported for single-cycle one-
dword or smaller transactions.
Table 41. High Performance Avalon-MM Master (HPRXM) Interface

Interface Name
rxm_write_o O Asserted by the core to

request a write to an
Avalon-MM slave.
rxm_address_o[avmm_add O The address of the Avalon-

r_width_hwtcl-1:0] MM slave being accessed.
rxm_writedata_o[255:0] O This bus contains the RX

data being written to the
slave.
rxm_byteenable_o[31:0] O These bits specify the valid

bytes for the write data. hprxm_master
rxm_burstcount_o[4:0] O The burst count, measured

in qwords, of the RX write or
read request. The maximum
amount of data in a burst is
512 bytes.
rxm_waitrequest_i I When asserted by the

external Avalon-MM slave,
this signal indicates that the
slave is not ready for the
next read or write request.
continued...
59
4. Interfaces
UG-20237 | 2021.02.04

Interface Name
rxm_read_o O Asserted by the core to

request a read.
rxm_readdata_i[255:0] I Read data returned from the

Avalon-MM slave in response
to a read request. This data
is sent to the IP core
through the TX interface.
rxm_readdatavalid_i I Asserted by the system

interconnect fabric to
indicate that the read data is
valid.
4.3.2.3. 32-Bit Control Register Access (CRA) Slave (Root Port only)
The CRA interface provides access to the control and status registers of the Avalon-MM
bridge. This interface has the following properties:
• 32-bit data bus
• Supports a single transaction at a time
• Supports single-cycle transactions (no bursting)
Note: When the Avalon-MM Hard IP for PCIe IP Core is in Root Port mode, and the
application logic issues a CfgWr or CfgRd via the CRA interface, it needs to fill the Tag
field in the TLP Header with the value 0x10 to ensure that the corresponding
Completion gets routed to the CRA interface correctly. If the application logic sets the
Tag field to some other value, the Avalon-MM Hard IP for PCIe IP Core does not
overwrite that value with the correct value.
Table 42. Avalon-MM CRA Slave Interface

Interface Name
cra_read_i I Read enable.
cra_write_i I Write request.
cra_address_i[14:0] I
cra_writedata_i[31:0] I Write data. The current

version of the CRA slave
interface is read-only.
Including this signal as a
part of the Avalon-MM
interface makes future
enhancements possible.
cra
cra_readdata_o[31:0] O Read data.
cra_byteenable_i[3:0] I Byte enable.
cra_waitrequest_o O Wait request to hold off

additional requests.
cra_chipselect_i I Chip select signal to this

slave.
cra_irq_o O Interrupt request. A port

request for an Avalon-MM
interrupt.
60
4. Interfaces
UG-20237 | 2021.02.04
4.4. Serial Data Interface

The P-Tile Avalon-MM IP for PCIe natively supports 4, 8, or 16 PCIe lanes. Each lane
includes a TX differential pair and an RX differential pair. Data is striped across all
available lanes.
Table 43. Serial Data Interface

Signal Name Direction Description
tx_p_out[<b>-1:0], O Transmit serial data outputs using the

tx_n_out[<b>-1:0] High Speed Differential I/O standard.
rx_p_in[<b>-1:0], I Receive serial data inputs using the

rx_n_in[<b>-1:0] High Speed Differential I/O standard.
Note: The value of the variable b depends on which configuration is active (1x16, 2x8 or
4x4).
• For 1x16, b = 16.

• For 2x8, b = 8.
• For 4x4, b = 4.
4.5. Hard IP Status Interface

This interface includes the signals that are useful for debugging, such as the link
status signal, LTSSM state outputs, etc. These signals are available when the optional
Power Management interface is enabled.
Table 44. Hard IP Status Interface

Signal Name Direction Description Clock Domain EP/RP
When asserted, this signal indicates the

link_up_o O p<n>_app_clk EP/RP
link is up.
When asserted, this signal indicates the

dl_up_o O p<n>_app_clk EP/RP
Data Link (DL) Layer is active.
ltssm_state_o[5:0 O Indicates the LTSSM state:

] • 6'h00: S_DETECT_QUIET
• 6'h01: S_DETECT_ACT
• 6'h02: S_POLL_ACTIVE
• 6'h03: S_POLL_COMPLIANCE
• 6'h04: S_POLL_CONFIG
• 6'h05: S_PRE_DETECT_QUIET
• 6'h06: S_DETECT_WAIT
• 6'h07: S_CFG_LINKWD_START
p<n>_app_clk EP/RP
• 6'h08: S_CFG_LINKWD_ACCEPT
• 6'h09: S_CFG_LANENUM_WAIT
• 6'h0A: S_CFG_LANENUM_ACCEPT
• 6'h0B: S_CFG_COMPLETE
• 6'h0C: S_CFG_IDLE
• 6'h0D: S_RCVRY_LOCK
• 6'h0E: S_RCVRY_SPEED
• 6'h0F: S_RCVRY_RCVRCFG
• 6'h10: S_RCVRY_IDLE
continued...
61
4. Interfaces
UG-20237 | 2021.02.04
• 6'h11: S_L0
• 6'h12: S_L0S
• 6'h13: S_L123_SEND_EIDLE
• 6'h14: S_L1_IDLE
• 6'h15: S_L2_IDLE
• 6'h16: S_L2_WAKE
• 6'h17: S_DISABLED_ENTRY
• 6'h18: S_DISABLED_IDLE
• 6'h19: S_DISABLED
• 6'h1A: S_LPBK_ENTRY
• 6'h1B: S_LPBK_ACTIVE
• 6'h1C: S_LPBK_EXIT
• 6'h1D: S_LPBK_EXIT_TIMEOUT
• 6'h1E: S_HOT_RESET_ENTRY
• 6'h1F: S_HOT_RESET
• 6'h20: S_RCVRY_EQ0
surprise_down_err O When active, indicates that a surprise

link down event is occurring. p<n>_app_clk RP
_o
4.6. Interrupt Interface

The P-Tile Avalon-MM IP for PCI Express supports Message Signaled Interrupts (MSI),
MSI-X interrupts, and legacy interrupts. MSI and legacy interrupts are mutually
exclusive.
Legacy interrupts, MSI, and MSI-X interrupts are all controlled and generated
externally to the Avalon-MM IP to ensure total flexibility of allocating interrupt
resources based on the user’s application needs.
To support domain-isolation, legacy interrupt messages, MSI, and MSI-X TLPs need to
be sent with the appropriate source IDs.
The following figure shows an example integrating an external interrupt controller with
the P-Tile Avalon-MM IP. The interrupt controller takes interrupt requests from the
external DMA controller as well as those from the user application.
62
4. Interfaces
UG-20237 | 2021.02.04
Figure 20. Example of an Interrupt Controller Integrated with an Endpoint P-Tile

Avalon-MM IP for PCI Express IP Core
P-Tile Avalon-MM IP for PCIe

User’s Interrupt Controller
Avalon-MM master
for MSI/MSI-X M S Bursting
generation Slave
PF & VF Numbers
Interrupt to VFm, n
Mapping
tl_cfg_ctl_o[15:0]
Interrupt to MSI/MSI-X tl_cfg_add_o[4:0]
Mapping
tl_cfg_func_o[2:0]
DMA Controller MSI-X Table

S M Bursting
Interrupts
Master
Interrupt Status Registers
BAR, PF & VF Numbers
Interrupts from
User’s Application intx_req_i[3:0]
4.6.1. Legacy Interrupts

If legacy interrupts are enabled at IP configuration time, the user’s interrupt controller
generates legacy interrupts by asserting the intx_req_i input signal which causes
the PCIe Hard IP to send the corresponding interrupt message. Use of legacy
interrupts to signal the completion of DMA transfers is not recommended as their
ordering with respect to the DMA traffic is not guaranteed.
4.6.2. MSI
If MSI or MSI-X are enabled at IP configuration time, the external interrupt controller
can generate MSI/MSI-X transactions by issuing memory writes to the Bursting Slave
or using the immediate write feature of the Write Data Mover, especially if signaling
the completion of a DMA transfer by the Write Data Mover. The interrupt controller
gets the address and data information to generate the MSI/MSI-X messages from the
MSI or MSI-X capability registers in the Transaction Layer in the P-Tile IP.
MSI interrupts are signaled on the PCI Express link using a single dword Memory Write
TLP. The user application issues an MSI request (MWr) through the Avalon-ST interface
and updates the configuration space register using the MSI interface.
For more details on the MSI Capability Structure, refer to Figure 55 on page 119.
The Mask Bits register and Pending Bits register are 32 bits in length each, with each
potential interrupt message having its own mask bit and pending bit. If bit[0] of the
Mask Bits register is set, interrupt message 0 is masked. When an interrupt message
is masked, the MSI for that vector cannot be sent. If software clears the mask bit and
the corresponding pending bit is set, the function must send the MSI request at that
time.
63
4. Interfaces
UG-20237 | 2021.02.04
You should obtain the necessary MSI information (such as the message address and
data) from the configuration output interface (tl_cfg_*) to create the MWr TLP in
the format shown below to be sent via the Avalon-ST interface.
Figure 21. Creating a MWr TLP for an MSI Request
MSI (Memory Write) Transaction

+0 +1 +2 +3
7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0
Fmt Type R T T E Attr AT 0 0 0 0Length
Byte 0 0 1 1 0 0 0 0 0 R TC R At tr H D P 00 0 0 0 0 0 1
Byte 4 Requester ID Tag Last DW First DW
0000 1111 Header
Byte 8 MSI Message Address [63:32]
Byte 12 MSI Message Address [31:0] 00
Byte 16 MSI Message Data 0000h Data
MSI Capability Structure

31 16 15 8 7 0
Message Control Next Capability Capability ID
Pointer (05h) DW0
Message Address [31:0] DW1
Message Address [63:32] DW2
Message Data DW3
Table 45. MSI Pending Bits Interface

Function number select for the

p<n>_app_cl
msi_pnd_func_i[2:0] I Pending Bits register in the MSI EP
capability structure.
k
Byte select for Pending Bits

Register in the MSI Capability
Structure. For example if
msi_pnd_addr_i[1:0] = 00,
bits [7:0] of the Pending Bits
register will be updated with
p<n>_app_cl
msi_pnd_addr_i[1:0] I EP
msi_pnd_byte_i[7:0]. If k
msi_pnd_addr_i[1:0] = 01,
bits [15:8] of the Pending Bits
register will be updated with
msi_pnd_byte_i[7:0].
Indicate that function has a p<n>_app_cl

msi_pnd_byte_i[7:0] I EP
pending associated message. k
64
4. Interfaces
UG-20237 | 2021.02.04
The following figure shows the timings of msi_pnd_* signals in three scenarios. The
first scenario shows the case when the MSI pending bits register is not used. The
second scenario shows the case when only physical function 0 is enabled and the MSI
pending bits register is used. The last scenario shows the case when four physical
functions are enabled and the MSI pending bits register is used.
Figure 22. Example Timing Diagrams for msi_pnd* Signals

p<n>_app_clk
msi_pnd_func_i[2:0] 0x0
msi_pnd_addr_i[1:0] 0x0
msi_pnd_byte_i[7:0] 0x0
p<n>_app_clk
msi_pnd_func_i[2:0] 0x0
msi_pnd_addr_i[1:0] 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3
msi_pnd_byte_i[7:0] B0 B1 B2 B3 B0 B1 B2 B3 B0 B1 B2 B3 B0 B1 B2 B3
p<n>_app_clk
msi_pnd_func_i[2:0] 0x0 0x1 0x0 0x1
msi_pnd_addr_i[1:0] 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3 0x0 0x1 0x2 0x3
msi_pnd_byte_i[7:0] B0 B1 B2 B3 B0 B1 B2 B3 B0 B1 B2 B3 B0 B1 B2 B3
There are 32 possible MSI messages. The number of messages requested by a

particular component does not necessarily correspond to the number of messages
allocated. For example, in the following figure, the Endpoint requests eight MSIs but is
only allocated two. In this case, you must design the Application Layer to use only two
allocated messages.
Figure 23. MSI Request Example
Root Complex
Endpoint Root CPU

Port
8 Requested
2 Allocated Interrupt
Block
Interrupt Register
The following table describes three example implementations. The first example
allocates all 32 MSI messages. The second and third examples only allocate 4
interrupts.
65
4. Interfaces
UG-20237 | 2021.02.04
Table 46. MSI Messages Requested, Allocated, and Mapped

MSI Allocated
32 4 4
System Error 31 3 3
Hot Plug and Power Management Event 30 2 3
Application Layer 29:0 1:0 2:0
MSI interrupts generated for Hot Plug, Power Management Events, and System Errors
always use Traffic Class 0. MSI interrupts generated by the Application Layer can use
any Traffic Class. For example, a DMA that generates an MSI at the end of a
transmission can use the same traffic control as was used to transfer data.
The following figure illustrates a possible implementation of the Interrupt Handler

Module with a per vector enable bit in the Application Layer. Alternatively, the
Application Layer could implement a global interrupt enable instead of this per vector
MSI.
Figure 24. Example Implementation of the Interrupt Handler Block

app_int_i
MSI info (from tl_cfg_ctl* /

Vector 0 Interrupt Enable 0 tl_cfg_addr* / tl_cfg_func*)
msi_req0 & Master Enable
R/W
Avalon-MM
IRQ Interrupt Request 0 single-dword MWR TLPs
Generation Arbitration &
TLP Generator msi_pnd_*
App Layer
Vector 1 Interrupt Enable 1

msi_req1
R/W
IRQ Interrupt Request 1

Generation
App Layer
4.6.3. MSI-X
The P-Tile Avalon-MM IP provides a Configuration Intercept Interface. User soft logic
can monitor this interface to get MSI-X Enable and MSI-X function mask related
information. User application logic needs to implement the MSI-X tables for all PFs and
VFs at the memory space pointed to by the BARs as a part of your Application Layer.
For more details on the MSI-X related information that you can obtain from the
Configuration Intercept Interface, refer to the MSI-X Registers section in the Registers
chapter.
66
4. Interfaces
UG-20237 | 2021.02.04
MSI-X is an optional feature that allows the user application to support large amount
of vectors with independent message data and address for each vector.
When MSI-X is supported, you need to specify the size and the location (BARs and
offsets) of the MSI-X table and PBA. MSI-X can support up to 2048 vectors per
function versus 32 vectors per function for MSI.
A function is allowed to send MSI-X messages when MSI-X is enabled and the function
is not masked. The application uses the Configuration Output Interface (address 0x0C
bit[5:4]) or Configuration Intercept Interface to access this information.
When the application needs to generate an MSI-X, it will use the contents of the MSI-X
Table (Address and Data) and generate a Memory Write through the Avalon-ST
interface.
You can enable MSI-X interrupts by turning on the Enable MSI-X option under the
PCI Express/PCI Capabilities tab in the parameter editor. If you turn on the
Enable MSI-X option, you should implement the MSI-X table structures at the
memory space pointed to by the BARs as a part of your Application Layer.
The MSI-X Capability Structure contains information about the MSI-X Table and PBA
Structure. For example, it contains pointers to the bases of the MSI-X Table and PBA
Structure, expressed as offsets from the addresses in the function's BARs. The
Message Control register within the MSI-X Capability Structure also contains the MSI-X
Enable bit, the Function Mask bit, and the size of the MSI-X Table. For a picture of the
MSI-X Capability Structure, refer to Figure 57 on page 120.
MSI-X interrupts are standard Memory Writes, therefore Memory Write ordering rules
apply.
Example:
Table 47. MSI-X Configuration

MSI-X Vector MSI-X Upper Address MSI-X Lower Address MSI-X Data
0 0x00000001 0xAAAA0000 0x00000001
1 0x00000001 0xBBBB0000 0x00000002
2 0x00000001 0xCCCC0000 0x00000003
Table 48. PBA Table

PBA Table PBA Entries
Offset 0 0x0
If the application needs to generate an MSI-X interrupt (vector 1), it will read the MSI-
X Table information, generate a MWR TLP through the Avalon-ST interface and assert
the corresponding PBA bits (bit[1]) in a similar fashion as for MSI generation.
The generated TLP will be sent to address 0x00000001_BBBB0000 and the data will
be 0x00000002. When the MSI-X has been sent, the application can clear the
associated PBA bits.
67
4. Interfaces
UG-20237 | 2021.02.04
4.6.3.1. Implementing MSI-X Interrupts

Section 6.8.2 of the PCI Local Bus Specification describes the MSI-X capability and
table structures. The MSI-X capability structure points to the MSI-X Table structure
and MSI-X Pending Bit Array (PBA) registers. The BIOS sets up the starting address
offsets and BAR associated with the pointer to the starting address of the MSI-X Table
and PBA registers.
1. Host software sets up the MSI-X interrupts in the Application Layer by completing
the following steps:
a. Host software reads the Message Control register at 0x050 register to
determine the MSI-X Table size. The number of table entries is the <value
read> + 1.
The maximum table size is 2048 entries. Each 16-byte entry is divided in 4
fields as shown in the figure below. For multi-function variants, BAR4 accesses
the MSI-X table. For all other variants, any BAR can access the MSI-X table.
The base address of the MSI-X table must be aligned to a 4 KB boundary.
Note that multi-function support is not available in the current release of Intel
Quartus Prime.
b. The host sets up the MSI-X table. It programs MSI-X address, data, and
masks bits for each entry as shown in the figure below.
Figure 25. Format of MSI-X Table

DWORD 3 DWORD 2 DWORD 1 DWORD 0 Host Byte Addresses
Vector Control Message Data Message Upper Address Message Address Entry 0 Base
Vector Control Message Data Message Upper Address Message Address Entry 1 Base + 1 × 16
Vector Control Message Data Message Upper Address Message Address Entry 2 Base + 2 × 16
Vector Control Message Data Message Upper Address Message Address Entry (N - 1) Base + (N - 1) × 16
c. The host calculates the address of the <nth> entry using the following
formula:
nth_address = base address[BAR] + 16<n>
2. When Application Layer has an interrupt, it drives an interrupt request to the IRQ
Source module.
3. The IRQ Processor reads the entry in the MSI-X table.
a. If the interrupt is masked by the Vector_Control field of the MSI-X table,
the interrupt remains in the pending state.
b. If the interrupt is not masked, IRQ Processor sends Memory Write Request to
the TX slave interface. It uses the address and data from the MSI-X table. If
Message Upper Address = 0, the IRQ Processor creates a three-dword
header. If the Message Upper Address > 0, it creates a 4-dword header.
4. The host interrupt service routine detects the TLP as an interrupt and services it.
Related Information
• Floor and ceiling functions
• PCI Local Bus Specification, Rev. 3.0
68
4. Interfaces
UG-20237 | 2021.02.04
4.7. Hot Plug Interface (RP Only)

Note: The Hot Plug interface is not supported in the 20.2 release of Intel Quartus Prime. It
may be supported in a future release.
Hot Plug support means that the device can be added to or removed from a system
during runtime. The Hot Plug Interface in the P-Tile Avalon-MM IP for PCIe allows an
Intel FPGA with this IP to safely provide this capability.
This section describes the signals reported by the on-board hot plug components in
the Downstream Port. This interface is available only if the Slot Status Register
of the PCI Express Capability Structure is enabled.
Refer to the Slot Status Register of the PCI Express Capability

Structure for additional information.
Table 49. Hot Plug Interface

sys_atten_button_pressed_i I Attention Button p<n>_app_clk RP

Pressed. Indicates
that the system
attention button was
pressed, and sets
the Attention Button
Pressed bit in the
Slot Status
Register.
sys_pwr_fault_det_i I Power Fault p<n>_app_clk RP

Detected. Indicates
the power controller
detected a power
fault at this slot.
sys_mrl_sensor_chged_i I MRL Sensor p<n>_app_clk RP

Changed. Indicates
that the state of the
MRL sensor has
changed.
sys_pre_det_chged_i I Presence Detect p<n>_app_clk RP

Changed. Indicates
that the state of the
card presence
detector has
changed.
sys_cmd_cpled_int_i I Command p<n>_app_clk RP

Completed Interrupt.
Indicates that the
Hot Plug controller
completed a
command.
sys_pre_det_state_i I Indicates whether or p<n>_app_clk RP

not a card is present
in the slot.
0 : slot is empty.
1 : card is present in
the slot.
continued...
69
4. Interfaces
UG-20237 | 2021.02.04
sys_mrl_sensor_state_i I MRL Sensor State. p<n>_app_clk RP

Indicates the state
of the manually
operated retention
latch (MRL) sensor.
0 : MRL is closed.
1 : MRL is open.
sys_eml_interlock_engaged_ I Indicates whether p<n>_app_clk RP

i the system
electromechanical
interlock is engaged,
and controls the
state of the
electromechanical
interlock status bit in
the Slot Status
Register.
sys_aux_pwr_det_i I Auxiliary Power p<n>_app_clk RP

Detected. Used to
report to the host
software that
auxiliary power
(Vaux) is present.
Refer to the Device
Status Register
in the PCI Express
Capability
Structure.
4.8. Power Management Interface

Note: The Power Management interface is not available in the 20.4 release of Intel Quartus
Prime. However, it may be available in a future release.
Software programs the device into a D-state by writing to the Power Management
Control and Status register in the PCI Power Management Capability
Structure. The power management output signals indicate the current power state.
The IP core supports the two mandatory power states: D0 (full power) and D3
(preparation for a loss of power). It does not support the optional D1 and D2 low-
power states.
The correspondence between the device power states (D states) and link power states
(L states) is as follows:
Table 50. Relationship Between Device and Link Power States

Device Power State Link Power State
D0 L0
D1 (not supported) L1
D2 (not supported) L1
D3 L1, L2/L3 Ready
P-Tile does not support ASPM.
70
4. Interfaces
UG-20237 | 2021.02.04
Table 51. Power Management Interface

Indicates the current power

pm_state_o[2:0] O p<n>_app_clk EP/RP
state.
Power management D-state

for each function.
x16/x8: • 0001b : D0
pm_dstate_o[31:0] • 0010b : D1
O Async EP/RP
x4: • 0100b : D2
pm_dstate_o[3:0] • 1000b : D3
• 0000b : uninitialized or
invalid
The application logic asserts

this signal for one cycle to
x16/x8: wake up the Power
apps_pm_xmt_pme_ Management Capability (PMC)
I p<n>_app_clk EP
i[7:0] state machine from a D1, D2,
x4: NA or D3 power state. Upon
wake-up, the IP core sends a
PM_PME message.
The application logic asserts

this signal to indicate that it is
ready to enter the L2/L3
Ready state. The
app_ready_entr_l23_i
x16/x8: signal is provided for
app_ready_entr_l2 applications that must control
I the L2/L3 Ready entry (in case p<n>_app_clk EP
3_i
certain tasks must be
x4: NA performed before going into
L2/L3 Ready). The core delays
sending PM_Enter_L23 (in
response to PM_Turn_Off) until
this signal becomes active.
This is a level-sensitive signal.
When these signals are

asserted, the P-Tile Avalon-MM
x16: IP will respond to
app_req_retry_en_ Configuration TLPs with a
i[7:0] Configuration Retry Status
x8: I (CRS) if it is not ready to Async EP
app_req_retry_en_ respond with non-CRS status
i since the last reset.
x4: NA For x4 ports, this signal is not
used and needs to be driven to
zero.
4.9. Configuration Output Interface

The Transaction Layer configuration output (tl_cfg) bus provides a subset of the
information stored in the Configuration Space. Use this information in conjunction with
the app_err* signals to understand TLP transmission problems.
71
4. Interfaces
UG-20237 | 2021.02.04
Table 52. Configuration Output Interface

Multiplexed data output from the

register specified by
tl_cfg_ctl_o[15:0 tl_cfg_add_o[4:0]. The
O p<n>_app_clk EP/RP
] detailed information for each field
in this bus is defined in the
following table.
This address bus contains the

index indicating which
tl_cfg_add_o[4:0] O Configuration Space register p<n>_app_clk EP/RP
information is being driven onto
the tl_cfg_ctl_o[15:0] bits.
Specifies the function whose

Configuration Space register values
are being driven out on
tl_cfg_ctl_o[15:0].
x16/x8: • 3'b000: Physical Function 0
tl_cfg_func_o[2:0 (PF0)
O p<n>_app_clk EP/RP
] • 3'b001: PF1
x4: NA and so on
In the 19.4 release of Intel
Quartus Prime, the P-Tile
Note:
Avalon-MM IP only supports
PF0.
The table below provides the tl_cfg_add_o[4:0] to tl_cfg_ctl_o[15:0]

mapping.
Table 53. Multiplexed Configuration Information Available on tl_cfg_ctl

tl_cfg_add_o[4:0] tl_cfg_ctl_o[15:8] tl_cfg_ctl_o[7:0]
[15]: memory space enable

[14]: IDO completion enable
[13]: perr_en Device control:
[7]: bus master enable
[12]: serr_en
5'h00 [6]: extended tag enable
[11]: fatal_err_rpt_en
[5:3]: maximum read request size
[10]: nonfatal_err_rpt_en
[2:0]: maximum payload size
[9]: corr_err_rpt_en
[8]: unsupported_req_rpt_en
[15]: IDO request enable

[14]: No Snoop enable
5'h01 bus number
[13]: Relaxed Ordering enable
[12:8]: Device number
[15]: pm_no_soft_rst [7:5]: reserved

[14]: RCB control [4]: system power control
5'h02 [13]: Interrupt Request (IRQ) disable [3:2]: system attention indicator
[12:8]: PCIe Capability IRQ message control
number [1:0]: system power indicator control
5'h03 Number of VFs [15:0]
[7]: ARI forward enable

[15]: reserved
[6]: Atomic request enable
5'h04 [14]: AtomicOP Egress Block field
[5:3]: TPH ST mode
(cfg_atomic_egress_block)
[2:1]: TPH enable
continued...
72
4. Interfaces
UG-20237 | 2021.02.04
[13:9]: ATS Smallest Translation Unit

(STU)[4:0] [0]: VF enable
[8]: ATS cache enable
[15:12]: auto negotiation link speed.

Link speed encoding values are:
• Gen1 : 0x1
• Gen2 : 0x2
5'h05
• Gen3 : 0x4
• Gen4 : 0x8
[11:1]: Index of Start VF [10:0]
[0]: reserved
5'h06 MSI Address [15:0]
5'h0A MSI Mask [15:0]
5'h0B MSI Mask [31:16]
[7]: Enable extended message data for

MSI (cfg_msi_ext_data_en)
[15]: cfg_send_f_err
[6]: MSI-X func mask
[14]: cfg_send_nf_err
5'h0C [5]: MSI-X enable
[13]: cfg_send_cor_err
[4:2]: Multiple MSI enable
[12:8]: AER IRQ message number [1]: 64-bit MSI
[0]: MSI enable
5'h0D MSI Data [15:0]
5'h0E AER uncorrectable error mask [15:0]
5'h0F AER uncorrectable error mask [31:16]
5'h10 AER correctable error mask [15:0]
5'h11 AER correctable error mask [31:16]
5'h12 AER uncorrectable error severity [15:0]
5'h13 AER uncorrectable error severity [31:16]
[7]: ACS function group enable

(cfg_acs_func_grp_en)
[6]: ACS direct translated P2P enable
(cfg_acs_p2p_direct_tranl_en)
[5]: ACS P2P egress control enable
(cfg_acs_egress_ctrl_en)
[4]: ACS upstream forwarding enable
[15:8]: ACS Egress Control Register (cfg_acs_up_forward_en)
5'h14
(cfg_acs_egress_ctrl_vec) [3]: ACS P2P completion redirect
enable
(cfg_acs_p2p_compl_redirect_en
)
[2]: ACS P2P request redirect enable
(cfg_acs_p2p_req_redirect_en)
[1]: ACS translation blocking enable
(cfg_acs_at_blocking_en)
continued...
73
4. Interfaces
UG-20237 | 2021.02.04
[0]: ACS source validation enable (RP)

(cfg_acs_validation_en)
[15]: reserved
[14]: 10-bit tag requester enable
(cfg_10b_tag_req_en)
[13]: VF 10-bit tag requester enable
(cfg_vf_10b_tag_req_en)
[12]: PRS_RESP_FAILURE [7:3]: reserved
5'h15 (cfg_prs_response_failure) [2:0]: ARI function group
(cfg_ari_func_grp)
[11]: PRS_UPRGI (cfg_prs_uprgi)
[10]: PRS_STOPPED
(cfg_prs_stopped)
[9]: PRS_RESET (cfg_prs_reset)
[8]: PRS_ENABLE (cfg_prs_enable)
PRS_OUTSTANDING_ALLOCATION
5'h16 (cfg_prs_outstanding_allocatio
n) [15:0]
PRS_OUTSTANDING_ALLOCATION
5'h17 (cfg_prs_outstanding_allocatio
n) [31:16]
[7]: Infinite credits for Posted header

[6]: Infinite credits for Posted data
[5]: Infinite credits for Completion
header
[15:10]: reserved [4]: Infinite credits for Completion data
[9]: Disable autonomous generation of [3]: End-end TLP prefix blocking
LTR clear message (cfg_end2end_tlp_pfx_blck)
5'h18
(cfg_disable_ltr_clr_msg) [2]: PASID enable
[8]: LTR mechanism enable (cfg_pf_pasid_en)
(cfg_ltr_m_en) [1]: Execute permission enable
(cfg_pf_passid_execute_perm_en
)
[0]: Privileged mode enable
(cfg_pf_passid_priv_mode_en)
[7]: Slot control power fault detect

enable (cfg_pwr_fault_det_en)
[6]: Slot control MRL sensor changed
enable (cfg_mrl_sensor_chged_en)
[5]: Slot control presence detect
changed enable
(cfg_pre_det_chged_en)
[15:9]: reserved [4]: Slot control hot plug interrupt
[8]: Slot control attention button enable (cfg_hp_int_en)
5'h19
pressed enable [3]: Slot control command completed
(cfg_atten_button_pressed_en) interrupt enable
(cfg_cmd_cpled_int_en)
[2]: Slot control DLL state change
enable (cfg_dll_state_change_en)
[1]: Slot control accessed
(cfg_hp_slot_ctrl_access)
[0]: PF’s SERR# enable
(cfg_br_ctrl_serren)
LTR maximum snoop latency register

5'h1A
(cfg_ltr_max_latency[15:0])
continued...
74
4. Interfaces
UG-20237 | 2021.02.04
LTR maximum no-snoop latency

5'h1B register
(cfg_ltr_max_latency[31:16])
[5:0]: auto negotiation link width

6’h01 = x1
[15:8]: enabled Traffic Classes (TCs) 6’h02 = x2
5'h1C
(cfg_tc_enable[7:0]) 6’h04 = x4
6’h08 = x8
6’h10 = x16
5'h1D MSI Data[31:16]
5'h1E N/A
5'h1F N/A
Note: The information on the Configuration Output (tl_cfg) bus is time-division

multiplexed (TDM).
• When tl_cfg_func[2:0] = 3'b000, tl_cfg_ctl[31:0] drive out the PF0

Configuration Space register values.
• Then, tl_cfg_func[2:0] are incremented to 3'b001.
• When tl_cfg_func[2:0] = 3'b001, tl_cfg_ctl[31:0] drive out the PF1
Configuration Space register values.
• This pattern repeats to cover all enabled PFs.
• The P-Tile Avalon-MM IP for PCIe only supports PF0.
Figure 26. Configuration Output Interface Timing Diagram

p<n>_app_clk
tl_cfg_add_0[4:0] 0x00 0x01 0x02 0x03 0x00 0x01 0x02
tl_cfg_ctl_0[15:0] PF0 DATA0 PF0 DATA1 PF0 DATA2 PF0 DATA3 PF1 DATA0 PF1 DATA1 PF1 DATA2
tl_cfg_func_0[2:0] 0 1
4.10. Hard IP Reconfiguration Interface

The Hard IP reconfiguration interface is an Avalon-MM slave interface with a 21-bit
address and an 8-bit data bus. It is also sometimes referred to as the User Avalon-MM
Interface. You can use this interface to dynamically modify the value of configuration
registers. Note that after a warm reset or cold reset, changes made to the
configuration registers of the Hard IP via the Hard IP reconfiguration interface are lost
as these registers revert back to their default values.
Note: This interface can be used in Endpoint and Root Port modes. It must be enabled if
Root Port mode is selected.
In Root Port mode, the application logic uses the Hard IP reconfiguration interface to
access its PCIe configuration space to perform link control functions (such as Hot
Reset, link disable, or link retrain).
75
4. Interfaces
UG-20237 | 2021.02.04
Table 54. Hard IP Reconfiguration Interface

Reconfiguration
clock
50 MHz - 125 MHz
hip_reconfig_clk I EP/RP
(Range)
100 MHz
(Recommended)
Avalon-MM read hip_reconfig_cl

hip_reconfig_readdata_o[7:0] O EP/RP
data outputs k
Avalon-MM read
data valid. When
asserted, the data
on
hip_reconfig_cl
hip_reconfig_readdatavalid_o O EP/RP
hip_reconfig_re k
addata_o[7:0] is
valid.
Avalon-MM write hip_reconfig_cl

hip_reconfig_write_i I EP/RP
enable k
Avalon-MM read hip_reconfig_cl

hip_reconfig_read_i I EP/RP
enable k
hip_reconfig_cl
hip_reconfig_address_i[20:0] I Avalon-MM address EP/RP
k
Avalon-MM write hip_reconfig_cl

hip_reconfig_writedata_i[7:0] I EP/RP
data inputs k
When asserted, this

signal indicates that
hip_reconfig_cl
hip_reconfig_waitrequest_o O the IP core is not EP/RP
ready to respond to
k
a request.
Reset signal. You

can tie it to ground
or leave it floating
dummy_user_avmm_rst I EP/RP
when using the Hard
IP Reconfiguration
Interface.
Reading and Writing to the Hard IP Reconfiguration Interface
Reading from the Hard IP reconfiguration interface of the P-Tile Avalon-MM IP for PCI
Express retrieves the current value at a specific address. Writing to the reconfiguration
interface changes the data value at a specific address. Intel recommends that you
perform read-modify-writes when writing to a register, because two or more features
may share the same reconfiguration address.
Modifying the PCIe configuration registers directly affects the behavior of the PCIe
device.
76
4. Interfaces
UG-20237 | 2021.02.04
Figure 27. Timing Diagram to Perform Read and Write Operations Using the Hard IP
Reconfiguration Interface
4.10.1. Address Map for the User Avalon-MM Interface

The User Avalon-MM interface provides access to the configuration registers and the IP
core registers. This interface includes an 8-bit data bus and a 21-bit address bus
(which contains the byte addresses).
There are two methods to access the configuration registers:

• Using direct User Avalon-MM interface (byte access)
• Using the Debug (DBI) register access (dword access). This method is useful when
you need to read/write the entire 32 bits at one time (Counter/ Lane Margining,
etc.)
The following diagram and table show the address offsets for physical function 0
(PF0), User Avalon-MM Port Configuration Register and Debug (DBI) Register.
77
4. Interfaces
UG-20237 | 2021.02.04
Figure 28. Address Map for the User Avalon-MM Interface

0x1fffff
0x104200 Debug_DBI_Data/Debug_DBI_Addr
0x104068 User Avalon-MM Port Configuration Register
0x001000
PF0 PCie Configuration Registers
0x000000
78
4. Interfaces
UG-20237 | 2021.02.04
Table 55. Configuration Space Offsets

Registers User Avalon-MM Offsets Comments
Physical function 0 0x0000 Refer to Appendix A for more details of

the PF configuration space. This PF is
available for x16, x8 and x4 cores.
User Avalon-MM Port Configuration 0x104068 Refer to User Avalon-MM Port

Register Configuration Register (Offset
0x104068) for more details.
Debug (DBI) Register 0x104200 to 0x104204 Refer to Using the Debug Register
Interface Access on page 79 for more
details.
4.10.2. Configuration Registers Access
4.10.2.1. Using Direct User Avalon-MM Interface (Byte Access)
Targeting PF Configuration Space Registers
User application needs to specify the offsets of the targeted PF registers.
For example, if the application wants to read the MSI Capability Register of PF0, it will
issue a Read with address 0x0050 to target the MSI Capability Structure of PF0.
Figure 29. PF Configuration Space Registers Access Timing Diagram

HIP reconfig clk
HIP reconfig addr 0x000050
HIP reconfig read
HIP reconfig readdata PF0 MSI Cap
HIP reconfig readdatavalid
Targeting VSEC Registers
User application needs to program the VSEC field (0x104068 bit[0]) first. Then all
accesses from the user Avalon-MM interface starting at offset 0xD00 will be translated
to VSEC configuration space registers.
Figure 30. VSEC Registers Access Timing Diagram

HIP reconfig clk
HIP reconfig addr 0x104068 0xD00
HIP reconfig write
HIP reconfig writedata 0x01
HIP reconfig read
HIP reconfig waitrequest
HIP reconfig readdata VSEC Cap
4.10.2.2. Using the Debug Register Interface Access
DEBUG_DBI_ADDR register is located at user Avalon-MM offsets 0x104204 to

0x104207 (corresponding to byte 0 to byte 3). For example, the d_done bit is bit 7 at
byte address 0x104207.
79
4. Interfaces
UG-20237 | 2021.02.04
Table 56. DEBUG_DBI_ADDR Register

Names Bits R/W Descriptions
d_done 31 RO 1: indicates debug DBI read/

write access done
d_write 30 R/W 1: write access

0: read access
d_warm_reset 29 RO 1: normal operation

0: warm reset is on-going
d_vf 28:18 R/W Specify the virtual function

number.
d_vf_select 17 R/W To access the virtual function

registers, set this bit to one.
d_pf 16:14 R/W Specify the physical function

number.
reserved 13:12 R/W Reserved
d_addr 11:2 R/W Specify the DW address for

the P-Tile Avalon-MM IP DBI
interface.
d_shadow_select 1 R/W Reserved. Clear this bit for

access to standard PCIe
configuration registers.
d_vsec_select 0 R/W If set, this bit allows access

to Intel VSEC registers.
DEBUG_DBI_DATA register is located at user Avalon-MM offsets 0x104200 to

0x104203 (corresponding to byte 0 to byte 3).
Table 57. DEBUG_DBI_DATA Register

Names Bits R/W Descriptions
d_data 31:0 R/W Read or write data for the P-

Tile Avalon-MM IP register
access.
To write all 32 bits in a Debug register at a time:

1. Use the user_avmm interface to access 0x104200 to 0x104203 to write the data
first.
2. Use the user_avmm interface to access 0x104204 to 0x104206 to set the address
and control bits.
3. Use the user_avmm interface to write to 0x104207 to enable the read/write bit
(bit[30]).
4. Use the user_avmm interface to access 0x104207 bit[31] to poll if the write is
complete.
80
4. Interfaces
UG-20237 | 2021.02.04
Figure 31. DBI Register Write Timing Diagram

HIP reconfig clk
HIP reconfig addr 0x104200 0x104201 0x104202 0x104203 0x104204 0x104205 0x104206 0x104207
HIP reconfig write
HIP reconfig writedata 0x01 0x23 0x45 0x67 ADDR CTRL 0x4
HIP reconfig read
HIP reconfig readdata 0x4 0xC
To read all 32 bits in a Debug register at a time:

1. Use the user_avmm interface to access 0x104204 to 0x104206 to set the address
and control bits.
2. Use the user_avmm interface to write to 0x104207 to enable the read bit
(bit[30]).
3. Use the user_avmm interface to access 0x104207 bit[31] to poll if the read is
complete.
4. Use the user_avmm interface to access 0x104200 to 0x104203 to read the data
Figure 32. DBI Register Read Timing Diagram

HIP reconfig clk
HIP reconfig addr 0x104204 0x104205 0x104206 0x104207 0x104207 0x104200 0x104201 0x104202 0x104203
HIP reconfig write
HIP reconfig writedata ADDR CTRL 0x0
HIP reconfig read
HIP reconfig readdata 0x0 0x8 D0 D1 D2 D3
4.11. PHY Reconfiguration Interface

The PHY reconfiguration interface is an optional Avalon-MM slave interface with a
26-bit address and an 8-bit data bus. Use this bus to read the value of PHY registers.
Refer to Table 62 on page 90 for details on addresses and bit mappings for the PHY
registers that you can access using this interface.
These signals are present when you turn on Enable PHY reconfiguration on the
Top-Level Settings tab using the parameter editor.
Please note that the PHY reconfiguration interface is shared among all the PMA quads.
Table 58. PHY Reconfiguration Interface

Reconfiguration clock
xcvr_reconfig_clk I 50 MHz - 125 MHz (Range) EP/RP
100 MHz (Recommended)
xcvr_reconfig_readdata[7:0 Avalon-MM read data xcvr_reconfig_

O EP/RP
] outputs clk
Avalon-MM read data valid.

xcvr_reconfig_readdatavali When asserted, the data on xcvr_reconfig_
O EP/RP
d xcvr_reconfig_readdat clk
a[7:0] is valid.
xcvr_reconfig_
xcvr_reconfig_write I Avalon-MM write enable EP/RP
clk
continued...
81
4. Interfaces
UG-20237 | 2021.02.04
Avalon-MM read enable.

This interface is not
pipelined. You must wait for
the return of the xcvr_reconfig_
xcvr_reconfig_read I EP/RP
xcvr_reconfig_readdat clk
a[7:0] from the current
read before starting another
read operation.
Avalon-MM address
[25:21] are used to indicate
the Quad.
5'b00001 : Quad 0
xcvr_reconfig_address[25:0 xcvr_reconfig_
I 5'b00010 : Quad 1 EP/RP
] clk
5'b00100 : Quad 2
5'b01000 : Quad 3
[20:0] are used to indicate
the offset address.
xcvr_reconfig_writedata[7: xcvr_reconfig_
I Avalon-MM write data inputs EP/RP
0] clk
When asserted, this signal

indicates that the PHY is not xcvr_reconfig_
xcvr_reconfig_waitrequest O EP/RP
ready to respond to a clk
request.
Reading from the PHY Reconfiguration Interface
Reading from the PHY reconfiguration interface of the P-Tile Avalon-MM IP for PCI
Express retrieves the current value at a specific address.
Figure 33. Timing Diagram to Perform Read Operations Using the PHY Reconfiguration
Interface
xcvr_reconfig_clk
xcvr_reconfig_address 0x000006
xcvr_reconfig_read
xcvr_reconfig_readdatavalid
xcvr_reconfig_readdata 0x01
xcvr_reconfig_waitrequest
82
UG-20237 | 2021.02.04
Send Feedback
5. Advanced Features
5.1. PCIe Port Bifurcation and PHY Channel Mapping

Note: Port bifurcation for Gen3 x8, Gen4 x8 and Gen4 x4 will be available in a future release
of Intel Quartus Prime.
The PCIe controller IP contains a set of port bifurcation muxes to remap the four
controller PIPE lane interfaces to the shared 16 PCIe PHY lanes. The table below shows
the relationship between PHY lanes and the port mapping.
Table 59. Port Bifurcation and PHY Channel Mapping

Bifurcation Mode Port 0 (x16) Port 1 (x8) Port 2 (x4) Port 3 (x4)
1 x16 0 - 15 NA NA NA
2 x8 0-7 8 - 15 NA NA
4 x4 4-7 8 - 11 0-3 12 - 15
Note: For more details on the bifurcation modes, refer to the Architecture section in chapter
2.
UG-20237 | 2021.02.04
Send Feedback
6. Troubleshooting/Debugging
As you bring up your PCI Express system, you may face issues related to FPGA
configuration, link training, BIOS enumeration, data transfer, and so on. This chapter
suggests some strategies to resolve the common issues that occur during bring-up.
You can additionally use the P-Tile Debug Toolkit to identify the issues.
6.1. Hardware
Typically, PCI Express link-up involves the following steps:
1. Link training
2. BIOS enumeration and data transfer
The following sections describe the flow to debug link issues during the hardware
bring-up. Intel recommends a systematic approach to diagnosing issues as illustrated
in the following figure.
Additionally, you can use the P-Tile Debug Toolkit for debugging the PCIe links when
using the P-Tile Avalon-MM IP for PCI Express. The P-Tile Debug Toolkit includes the
following features:
• Protocol and link status information.
• Basic and advanced debugging capabilities including PMA register access and Eye
viewing capability.
UG-20237 | 2021.02.04
Figure 34. PCI Express Debug Flow Chart
Is the link
Go to “6.1.1 Debugging
Start training Link training issues”
successful? No
System Reset
Yes
Is the data Go to “6.1.2 Debugging data

transfer transfer and
successful? No performance issues”
Yes
End
6.1.1. Debugging Link Training Issues

The Physical Layer automatically performs link training and initialization without
software intervention. This is a well-defined process to configure and initialize the
device's Physical Layer and link so that PCIe packets can be transmitted.
Some examples of link training issues include:

• Link fails to negotiate to expected link speed.
• Link fails to negotiate to the expected link width.
• LTSSM fails to reach/stay stable at L0.
Flow Chart for Debugging Link Training Issues
Use the flow chart below to identify the potential cause of the issue seen during link
training when using the P-Tile Avalon-MM IP for PCI Express.
85
UG-20237 | 2021.02.04
Figure 35. Link Training Debugging Flow

Begin
A.Observation: Frequent transitions on ltssm_state_o

Does link signal between L0 and Recovery.rcvrlock states
Does the go to L0 at
LTSSM advertised speed with Issue: Signal Integrity issues (or) sub optimal EQ settings
enter L0? Yes frequent recoveries? Yes Resolution: Redo the Equalization (*)
No A.Observation: Wrong lane numbers encoded in

TS1/TS2 (Observed using Protocol Analyzer)
Issue: Improper lane reversal
Resolution: Check the lane routing
Does the
link train to OR
No L0 with reduced B.Observation: Timeout during EQ Phases on few
lane width? Yes lanes when monitoring ltssm_state_o signal
Issue: Signal Integrity issues/Sub optimal EQ settings
on few lanes
Resolution: Redo the Equalization (*)
No
A.Observation: Loop of Detect.Quiet

–> Detect.Active –> Polling.Active
–> Recovery.rcvrlock transitions observed on
Does the link ltssm_state_o signal
End train to L0 at Issue: Poor refclk quality
No reduced speed? Yes Resolution: Check the reference clock quality is good.
(e.g. Jitter, phase noise, etc). Ensure that the clocks
used are in accordance with the guidelines described in
the User Guide
OR
B.Observation: Timeout during EQ Phases on few
lanes when monitoring ltssm_state_o signal
Is the link Yes Is there
receiver detected Issue: Signal Integrity issues/Sub optimal EQ settings
out of
at the far end? on few lanes
reset?
Resolution: Redo the Equalization (*)
Yes
No No
Observation: ltssm_state_o signal Observation: ltssm_state_o signal toggles Observation: ltssm_state_o signal
stuck at Detect.Quiet state between Detect.Quiet and Detect.Active. Check the transitions from Detect.Quiet –>
Receiver detection status from the registers for Detect.Active –> Polling.Active –>
Issue: IP is in reset state
successful receiver detection Polling.Compliance states.
Resolution: Check if the pin_perst_n
Issue: Far end receiver not detected by the FPGA TX Issue: Far end device failing receiver detection
reset signal is in reset
Resolution: Check coupling capacitance, Resolution: Check far end coupling capacitance,
far end termination resistance and TX OCT values are near end termination resistance and TX OCT values
in accordance to the spec are in accordance to the spec
Note: (*) Redo the equalization using the Link Equalization Request 8.0 GT/s bit
of the Link Status 2 register for 8.0 GT/s or Link Equalization Request 16.0 GT/s
bit of the 16.0 GT/s Status Register.
Use the following debug tools for debugging link training issues observed on the PCI
Express link when using the P-tile Avalon-MM IP for PCI Express.
86
UG-20237 | 2021.02.04
6.1.1.1. Generic Tools and Utilities
You can use utilities like lspci, setpci to obtain general information of the device like
link speed, link width etc.
Example: To read the negotiated link speed for the P-Tile device in a system, you can
use the following commands:
sudo lspci –s $bdf -vvv
-s refers to “slot” and is used with the bus/device/function number (bdf) information.
Use this command if you know the bdf of the device in the system topology.
sudo lspci –d <1172>:$did -vvv
-d refers to device and is used with the device ID (vid:did). Use this command to
search using the device ID.
Figure 36. lspci Output
The LnkCap under Capabilities indicates the advertised link speed and width
capabilities of the device. The LnkSta under Capabilities indicates the negotiated
link speed and width of the device.
6.1.1.2. SignalTapII Logic Analyzer
Using the SignalTapII Logic Analyzer, you can monitor the following top-level signals
from the P-Tile Avalon-MM IP for PCI Express to confirm the failure symptom for any
port type (Root port, Endpoint or TLP Bypass) and configuration (Gen4/Gen3).
87
UG-20237 | 2021.02.04
Table 60. Top-Level Signals to be Monitored for Debugging

Signals Description Expected Value for Successful Link-
up
pin_perst_n Active-low asynchronous input signal 1'b1

to the PCIe Hard IP.
Implements the PERST# function
defined by the PCIe specification.
p0_reset_status_n Active-low output signal from the PCIe 1'b1

Hard IP, synchronous to
p<n>_app_clk.
Held low until pin_perst_n is
deasserted and the PCIe Hard IP
comes out of reset, synchronous to
p<n>_app_clk.
When port bifurcation is used, there is
one such signal for each Avalon-MM
interface.
ninit_done Active-low output signal from the PCIe 1'b0

Hard IP. High indicates that the FPGA
device is not yet fully configured, and
low indicates the device has been
configured and is in normal operating
mode.
link_up_o Active-high output signal from the PCIe 1'b1

p<n>_app_clk.
Indicates that the Physical Layer link is
up.
dl_up_o Active-high output signal from the PCIe 1'b1

p<n>_app_clk.
Indicates that the Data Link Layer is
active.
ltssm_state_o[5:0] Indicates the LTSSM state, 6'h11 (L0)

synchronous to p<n>_app_clk.
6.1.1.3. Additional Debug Tools
Use the Hard IP reconfiguration interface and PHY reconfiguration interface on the P-
Tile Avalon-MM IP for PCI Express to access additional registers (for example, receiver
detection, lane reversal etc.).
Figure 37. Register Access for Debug
PHY
PCIe Controllers
Port N
PLLA PMA x16
PCIe PCIe MAC DLL TL Hard IP Reconfig
PLLB Quad N
x16 Lanes PCS Interface
PHY Registers
Registers
PHY Reconfig
Interface
Using the Hard IP Reconfiguration Interface
88
UG-20237 | 2021.02.04
Refer to the section Hard IP Reconfiguration Interface for details on this interface and
the associated address map.
The following table lists the address offsets and bit settings for the PHY status
registers. Use the Hard IP Reconfiguration Interface to access these read-only
registers.
Table 61. Hard IP Reconfiguration Interface Register Map for PHY Status
Offset Bit Position Register
0x0003E9 [0] RX polarity
[1] RX detection
[2] RX Valid
[3] RX Electrical Idle
[4] TX Electrical Idle
0x0003EC [7] Framing error
0x0003ED [7] Lane reversal
Follow the steps below to access registers in Table 61 on page 89 using the Hard IP
reconfiguration interface:
1. Enable the Hard IP reconfiguration interface (User Avalon-MM interface) using the
IP Parameter Editor.
2. Set the lane number for which you want to read the status by performing a read-
modify-write to the address hip_reconfig_addr_i[20:0] with write data of
lane number on hip_reconfig_writedata_i[7:0] using the Hard IP
reconfiguration interface signals.
• hip_reconfig_write_i = 1’b1
• hip_reconfig_addr_i[20:0] = 0x0003E8
• hip_reconfig_writedata_i[3:0] = <Lane number>, where Lane number
= 4’h0 for lane 0, 4’h1 for lane 1, 4’h2 for lane 2, …
3. Read the status of the register you want by performing a read operation from the
address hip_reconfig_addr_i[20:0] using the Hard IP reconfiguration
interface signals.
• hip_reconfig_read_i = 1’b1
• hip_reconfig_addr_i[20:0] = <offset>
Offset = Refer to Table 61 on page 89 for the offset mapping.
• hip_reconfig_readdata_o[7:0] = Refer to Table 61 on page 89 for the
bit position mapping.
Example 1: To read the RX detection status of Lane0 using the registers

1. Enable the Hard IP reconfiguration interface using the IP Parameter Editor.
2. Perform read-modify-write to address 0x0003E8 to set the lane number to 0 using
the Hard IP reconfiguration interface signals.
89
UG-20237 | 2021.02.04
• hip_reconfig_write_i = 1’b1
• hip_reconfig_writedata_i[3:0] = 4'h0
3. Read the status of the RX detection register by performing a read operation from
the address 0x0003E9[1] using the Hard IP reconfiguration interface signals.
• hip_reconfig_read_i = 1’b1
• hip_reconfig_readdata_o[1] = 1'b1 (Far end receiver detected)
Using the PHY Reconfiguration Interface
Refer to the section PHY Reconfiguration Interface for details on how to use this
interface.
Follow the steps below to access registers in Table 62 on page 90 using the PHY
reconfiguration interface.
1. Enable the PHY reconfiguration interface using the IP Parameter Editor.
2. Set the Quad and address offset from which you want to read the status by
performing a read operation from the address xcvr_reconfig_addr_i[25:0]
using the PHY reconfiguration interface signals.
• xcvr_reconfig_read_i = 1’b1
• xcvr_reconfig_addr_i[25:0] = {5-bit Quad mapping, 21-bit address
offset}. Refer to Table 62 on page 90 for the address offset and bit mapping.
• xcvr_reconfig_readdata_o[7:0] = Refer to Table 62 on page 90 for the
address offset and bit mapping.
Table 62. PHY Reconfiguration Interface Register Map for PHY Status
PHY Offset Bit Position Register
0x000006 [7] PLLA state output status signal.

1'b1 indicates that PLLA is locked.
0x00000a [7] PLLB state output status signal.

1'b1 indicates that PLLB is locked.
Example 2: To read the PLLA status using the registers

1. Enable the PHY reconfiguration interface using the IP Parameter Editor.
2. Perform a read from address 0x000006 to read the PLLA status output of Quad0
using the PHY reconfiguration interface signals.
• xcvr_reconfig_read_i = 1'b1
• xcvr_reconfig_addr_i[25:0] = 0x000006
• xcvr_reconfig_readdata_o[7:0] = 8'h80
• xcvr_reconfig_readdata_i = 1'b1 (PLLA state output high indicating PLL
lock)
90
UG-20237 | 2021.02.04
6.1.2. Debugging Data Transfer and Performance Issues

There are many possible reasons causing the PCIe link to stop transmitting data. The
PCI Express base specification defines three types of errors, outlined in the table
below:
Table 63. Error Types Defined by the PCI Express Base Specification
Type Responsible Agent Description
Correctable Hardware While correctable errors may affect

system performance, data integrity is
maintained.
Uncorrectable, non-fatal Device software Uncorrectable, non-fatal errors are

defined as errors in which data is lost,
but system integrity is maintained. For
example, the fabric may lose a
particular TLP, but it still works without
problems.
Uncorrectable, fatal System software Errors generated by a loss of data and

system failure are considered
uncorrectable and fatal. Software must
determine how to handle such errors:
whether to reset the link or implement
other means to minimize the problem.
Table 64. Correctable Error Status Register (AER)

Observation Issue Resolution
Receiver error bit set Physical layer error which may be due Use the Hard IP reconfiguration
to a PCS error when a lane is in L0, or interface and the flow chart in Figure
a Control symbol being received in the 35 on page 86 to obtain more
wrong lane, or signal Integrity issues information about the error.
where the link may transition from L0
to the Recovery state.
Bad DLLP bit set Data link layer error which may occur Use the Hard IP reconfiguration
when a CRC verification fails. interface to obtain more information
about the error.
Bad TLP bit set Data link layer error which may occur Use the Hard IP reconfiguration
when an LCRC verification fails or when interface to obtain more information
a sequence number error occurs. about the error.
Replay_num_rollover bit set Data link layer error which may be due Use the Hard IP reconfiguration
to TLPs sent without success (no ACK) interface to obtain more information
four times in a row. about the error.
replay timer timeout status bit set Data link layer error which may occur Use the Hard IP reconfiguration
when no ACK or NAK was received interface to obtain more information
within the timeout period for the TLPs about the error.
transmitted.
Advisory non-fatal Transaction layer error which may be

due to higher priority uncorrectable
error detected.
Corrected internal error bits set Transaction layer error which may be Use the Hard IP reconfiguration
due to an ECC error in the internal interface and DBI registers to obtain
Hard IP RAM. more information about the error.
91
UG-20237 | 2021.02.04
Table 65. Uncorrectable Error Status Register (AER)

Data link protocol error Data link layer error which may be due Use the Hard IP reconfiguration
to transmitter receiving an ACK/NAK interface to obtain more information
whose Seq ID does not correspond to about the error.
an unacknowledged TLP or ACK
sequence number.
Surprise down error Data link layer error which may be due Use the Hard IP reconfiguration
to link_up_o getting deasserted interface and DBI registers to obtain
during L0, indicating the physical layer more information about the error.
link is going down unexpectedly.
Flow control protocol error Transaction layer error which can be Use the TX/RX flow control interface,
due to the receiver reporting more Hard IP reconfiguration interface to
than the allowed credit limit. obtain more information about the
This error occurs when a component error.
does not receive updated flow control
credits with the 200 μs limit.
Poisoned TLP received Transaction layer error which can be Use the Hard IP reconfiguration
due to a received TLP with the EP bit interface to obtain more information on
set. the error and determine the
appropriate action.
Completion timeout Transaction layer error which can be Use the Hard IP reconfiguration
due to a completion not received within interface to obtain more information on
the required amount of time after a the error.
non-posted request was sent.
Completer abort Transaction layer error which can be Use the Hard IP reconfiguration
due to a completer being unable to interface to obtain more information on
fulfill a request due to a problem with the error.
the requester or a failure of the
completer.
Unexpected completion Transaction layer error which can be Use the Hard IP reconfiguration
due to a requester receiving a interface to obtain more information on
completion that doesn’t match any the error.
request awaiting a completion.
The TLP is deleted by the Hard IP and
not presented to the Application Layer.
Receiver overflow Transaction layer error which can be Use the TX/RX flow control interface
due to a receiver receiving more TLPs and Hard IP reconfiguration interface
than the available receive buffer space. to obtain more information on the
The TLP is deleted by the Hard IP and error.
Malformed TLP Transaction layer error which can be Use the Hard IP reconfiguration
due to errors in the received TLP interface to obtain more information on
header. the error.
The TLP is deleted by the Hard IP and
ECRC error Transaction layer error which can be Use the Hard IP reconfiguration
due to an ECRC check failure at the interface to obtain more information on
receiver despite the fact that the TLP is the error.
not malformed and the LCRC check is
valid.
The Hard IP block handles this TLP
automatically. If the TLP is a non-
posted request, the Hard IP block
generates a completion with a
completer abort status. The TLP is
deleted by the Hard IP and not
presented to the Application Layer.
continued...
92
UG-20237 | 2021.02.04
Unsupported request Transaction layer error which can be Use the Hard IP reconfiguration
due to the completer being unable to interface to obtain more information on
fulfill the request. the error.
The TLP is deleted in the Hard IP block
and not presented to the Application
Layer. If the TLP is a non-posted
request, the Hard IP block generates a
completion with Unsupported Request
status.
ACS violation Transaction layer error which can be Use the Hard IP reconfiguration
due to access control error in the interface to obtain more information on
received posted or non-posted request. the error.
Uncorrectable internal error Transaction layer error which can be Use the Hard IP reconfiguration
due to an internal error that cannot be interface and DBI registers to obtain
corrected by the hardware. more information on the error.
Atomic egress blocked Use the Hard IP reconfiguration

interface to obtain more information on
the error.
TLP prefix blocked EP or RP only Use the Hard IP reconfiguration

the error.
Poisoned TLP egress blocked EP or RP only Use the Hard IP reconfiguration

the error.
Use the debug tools mentioned in the next two sections for debugging link training
issues observed on the PCI Express link when using the P-Tile Avalon-MM IP for PCI
Express.
6.1.2.1. Advanced Error Reporting (AER)
Each PCI Express compliant device must implement a basic level of error management
and can optionally implement advanced error management. The PCI Express
Advanced Error Reporting Capability is an optional Extended Capability that may be
implemented by PCI Express device functions supporting advanced error control and
reporting.
The P-Tile Avalon-MM IP for PCI Express implements both basic and advanced error
reporting. Error handling for a Root Port is more complex than that of an Endpoint. In
this P-Tile Avalon-MM IP for PCI Express, the AER capability is enabled by default.
Use the AER capability of the PCIe Hard IP to identify the type of error and the
protocol stack layer in which the error may have occurred. Refer to the PCI Express
Capability Structures section of the Configuration Space Registers appendix for the
AER Extended Capability Structure and the associated registers.
6.1.2.2. Second-Level Debug Tools
Use the following debug tools for second-level debug of any issue observed on the PCI
Express link when using P-Tile:
Using the Hard IP Reconfiguration Interface

• Refer to the section Hard IP Reconfiguration Interface on page 75 for details on
this interface and the address map.
93
UG-20237 | 2021.02.04
Using the PHY Reconfiguration Interface

• Refer to the section PHY Reconfiguration Interface on page 81 for details on this
interface and the address map.
6.2. Debug Toolkit
6.2.1. Overview
The P-Tile Debug Toolkit is a System Console-based tool for P-Tile that provides real-
time control, monitoring and debugging of the PCIe links at the Physical, Data Link
and Transaction layers.
The P-Tile Debug Toolkit allows you to:

• View protocol and link status of the PCIe links per port.
• View PLL and per-channel status of the PCIe links per port.
• Control the channel analog settings.
• View the receiver eye and measure the eye height and width.
• Indicate the presence of a re-timer connected between the link partners.
The following figure provides an overview of the P-Tile Debug Toolkit in the P-Tile
Avalon-MM IP for PCI Express.
Figure 38. Overview of the P-Tile Debug Toolkit
intel_pcie_ptile_avmm
P-Tile Debug Toolkit
PCle Config Space Registers

AVMM (Port 0)
..
.
hip-reconfig_* PCle Config Space Registers
AVMM (Port 3)
NPDME
AVMM PHY Registers (Quad 0)

..
System
.
xcvr_reconfig_*
Console
GUI AVMM PHY Registers (Quad 3)
When you enable the P-Tile Debug Toolkit, the intel_pcie_ptile_avmm module of
the generated IP includes the Debug Toolkit modules and related logic as shown in the
figure above.
Drive the Debug Toolkit from a System Console. The System Console connects to the
Debug Toolkit via an Native PHY Debug Master Endpoint (NPDME). Make this
connection via an Intel FPGA Download Cable.
94
UG-20237 | 2021.02.04
This PHY reconfiguration interface clock (xcvr_reconfig_clk) is used to clock the

following interfaces:
• The NPDME module
• PHY reconfiguration interface (xcvr_reconfig)
• Hard IP reconfiguration interface (hip_reconfig)
Provide a clock source (50 MHz - 125 MHz, 100 MHz recommended clock frequency) to
drive the xcvr_reconfig_clk clock. Use the output of the Reset Release Intel FPGA
IP to drive the ninit_done, which provides the reset signal to the NPDME module.
Note: When using the port bifurcation feature, always connect the xcvr_reconfig_clk of
Port0 to a clock source. This signal is used to provide the clock to the Debug Toolkit.
Note: When you enable the P-Tile Debug Toolkit, the Hard IP reconfiguration interface is
enabled by default.
When you run a dynamically-generated design example on the Intel Development Kit,
make sure that clock and reset signals are connected to their respective sources and
appropriate pin assignments are made. Here are some sample .qsf assignments for
the Debug Toolkit for Intel Stratix 10 DX devices:
• set_location_assignment PIN_A31 -to p0_hip_reconfig_clk_clk
• set_location_assignment PIN_C23 -to xcvr_reconfig_clk_clk
6.2.2. Enabling the P-Tile Debug Toolkit

To enable the P-Tile Debug Toolkit in your design, enable the option Enable Debug
Toolkit in the PCIe Configuration, Debug and Extension options tab of the Intel
FPGA P-Tile Avalon-MM IP for PCI Express.
When using bifurcated ports, you can enable the Debug Toolkit for each bifurcated
port by enabling the option Enable Debug Toolkit on each of the bifurcated ports.
Note: When you enable the P-Tile Debug Toolkit in the IP, the Hard IP reconfiguration
interface and the PHY reconfiguration interface will be used by the Debug Toolkit.
Hence, you will not be able to drive logic on these interfaces from the FPGA fabric.
6.2.3. Launching the P-Tile Debug Toolkit

Use the design example you compiled by following the Quick Start Guide to familiarize
yourself with the P-Tile Debug Toolkit. Follow the steps in the Generating the Design
Example and Compiling the Design Example to generate the SRAM Object File, (.sof)
for this design example.
To use the P-Tile Debug Toolkit, download the .sof to the Intel Development Kit. Then,
open the System Console and load the design to the System Console as well. Loading
the .sof to the System Console allows the System Console to communicate with the
design using NPDME. NPDME is a JTAG-based Avalon-MM master. It drives Avalon-MM
slave interfaces in the PCIe design. When using NPDME, the Intel Quartus Prime
software inserts the debug interconnect fabric to connect with JTAG.
Here are the steps to complete these tasks:
95
UG-20237 | 2021.02.04
1. Use the Intel Quartus Prime Programmer to download the .sof to the Intel FPGA
Development Kit.
Note: To ensure correct operation, use the same version of the Intel Quartus
Prime Programmer and Intel Quartus Prime Pro Edition software that you
used to generate the .sof.
2. To load the design into System Console:
a. Launch the Intel Quartus Prime Pro Edition software.
b. Start System Console by choosing Tools, then System Debugging Tools,
then System Console.
c. On the System Console File menu, select Load design and browse to the .sof
file.
d. Select the .sof and click OK. The .sof loads to the System Console.
3. The System Console Toolkit Explorer window will list all the DUTs in the design
that have the P-Tile Debug Toolkit enabled.
a. Select the DUT with the P-Tile Debug Toolkit you want to view. This will open
the Debug Toolkit instance of that DUT in the Details window.
96
UG-20237 | 2021.02.04
b. Click on the ptile_debug_toolkit_avmm to open that instance of the Toolkit.

Once the Debug Toolkit is initialized and loaded, you will see the following
message in the Messages window: “Initializing P-Tile debug toolkit –
done”.
c. A new window Main view will open with a view of all the channels in that
instance.
97
UG-20237 | 2021.02.04
6.2.4. Using the P-Tile Debug Toolkit

The following sections describe the different tabs and features available in the Debug
Toolkit.
A. Main View
The main view tab lists a summary of the transmitter and receiver settings per
channel for the given instance of the PCIe IP.
The following table shows the channel mapping when using bifurcated ports.
Table 66. Channel Mapping for Bifurcated Ports

Toolkit Channel X16 Mode 2X8 Mode 4x4 Mode
Lane 0 Lane 0 Lane 0 Lane 0

continued...
98
UG-20237 | 2021.02.04
Toolkit Channel X16 Mode 2X8 Mode 4x4 Mode
B. Toolkit Parameters
The Toolkit parameters window has 2 sub-tabs.
B.1. P-Tile Information
This lists a summary of the P-Tile PCIe IP parameter settings in the PCIe IP Parameter
Editor when the IP was generated, as read by the P-Tile Debug Toolkit when initialized.
When using bifurcated ports, you will see all the P-Tile information for each port for
which the Debug Toolkit has been enabled.
All the information is read-only.
Use the Get P-tile Info button to read the settings.
Table 67. P-Tile Available Parameter Settings

Parameter Values Descriptions
Indicates the Vendor ID as set in the IP

Intel Vendor ID 1172
Parameter Editor.
Protocol PCIe Indicates the Protocol.
HIP Type Root Port, End Point Indicates the Hard IP Port type.
intel_pcie_ptile_ast,
Intel IP Type Indicates the IP type used.
intel_pcie_ptile_avmm
Indicates the advertised speed as

Advertised speed Gen3, Gen4
configured in the IP Parameter Editor.
Indicates the advertised width as

Advertised width x16, x8, x4
configured in the IP Parameter Editor.
Indicates the negotiated speed during

Negotiated speed Gen3, Gen4
link training.
Indicates the negotiated link width

Negotiated width x16, x8, x4
during link training.
Link status Link up, link down Indicates if the link (DL) is up or not.
Indicates if a retimer was detected

Retimer 1 Detected, not detected between the Root Port and the
Endpoint.
Indicates if a retimer was detected

Retimer 2 Detected, not detected between the Root Port and the
Endpoint.
99
UG-20237 | 2021.02.04
Figure 39. Example of P-Tile Parameter Settings
B.2. PCIe Configuration Space
This lists a summary of the P-Tile PCIe configuration settings of the PCIe configuration
space registers, as read by the P-Tile Debug Toolkit when initialized.
All the information is read-only.
Use the Read cfg space button to read the settings.
100
UG-20237 | 2021.02.04
Figure 40. Example of P-Tile PCIe Configuration Settings
C. Channel Parameters
The channel parameters window allows you to monitor and control the transmitter and
receiver settings for a given channel. It has the following 2 sub-windows.
C.1. TX Path
This tab allows you to monitor and control the transmitter settings for the channel
selected. Use the TX Refresh button to read the settings, TX Apply Ch to apply the
settings to the selected channel, and TX apply all to apply the settings to all
channels.
Table 68. Transmitter Settings

Parameters Values Descriptions
Indicates reference clock is

enabled for the PHY.
Enable: Reference clock is
Refclk enable Enable, Disable
enabled for the PHY.
Disable: Reference clock is
PHY Status disabled for the PHY.
Indicates the PHY is in reset

mode.
PHY reset Normal, Reset
Normal: PHY is out of reset.
Reset: PHY is in reset.
Indicates if TX lane is
TX Status TX Lane enable Enable, Disable
enabled in the PHY.
continued...
101
UG-20237 | 2021.02.04
Enable: TX lane is enabled in

the PHY.
Disable: TX lane is disabled
in the PHY.
Indicates if TX driver is
enabled and serial data is
transmitted.
Enable: TX driver for the
TX Data enable Enable, Disable corresponding lane is
enabled.
Disable: TX driver for the
corresponding lane is
disabled.
Indicates if TX (TX datapath,

TX settings) is in reset or
normal operating mode.
TX Reset Normal, Reset
Normal: TX is in normal
operating mode.
Reset: TX is in reset.
Indicates if the TX PLL is

powered on or powered
down. This is dependent on
the PLL selected as indicated
by TX PLL select.
There is one set of PLLs per
Quad. The TX path of each
channel reads out the PLL
status corresponding to that
Quad.
• TX path for Ch0 to 3:
TX PLL enable Enable, Disable Status of PLLs in Quad0
Status of PLLs in Quad1
Enable: TX PLL is powered
TX PLL
on.
Disable: TX PLL is powered
down.
Indicates which PLL is

selected.
Quad.
PLLA: Gen1/Gen2
TX PLL select • TX path for Ch0 to 3:
PLLB: Gen3/Gen4 Status of PLLs in Quad0
continued...
102
UG-20237 | 2021.02.04
Indicates if TX PLL is locked.

This is dependent on the PLL
selected as indicated by TX
PLL select.
Quad.
TX PLL lock Green, Red • TX path for Ch0 to 3:
Green: TX PLL is locked.
Red: TX PLL is not locked.
Indicates the transmitter

Gen3: 15 current boost level when the
Iboost level
Gen4: 15 TX amplitude boost mode is
enabled.
Indicates if the TX swing

boost level is enabled.
TX VOD Gen3 Enable Enable: TX swing boost is
Vboost en
Gen4 Enable enabled.
Disable: TX swing boost is
disabled.
Gen3: 5 Indicates the TX Vboost

Vboost level
Gen4: 5 level.
Gen3: 20 (Preset 8) Indicates transmitter driver

Pre-shoot coefficient output pre-emphasis (pre-
Gen4: 0 (Preset 0) shoot coefficient).

TX Equalization Main coefficient output pre-emphasis (main
Gen4: 30 (Preset 0) coefficient).

Post coefficient output pre-emphasis (post
Gen4: 40 (Preset 0) coefficient).
103
UG-20237 | 2021.02.04
Figure 41. Example of Transmitter Settings
C.1. RX Path
This tab allows you to monitor and control the receiver settings for the channel
selected. Use the RX Refresh button to read the settings, RX Apply Ch to apply the
settings to the selected channel, and RX apply all to apply the settings to all
channels.
Table 69. Receiver Settings

Indicates if RX lane is
enabled in the PHY.
Enable: RX lane is enabled
RX Lane enable Enable, Disable
in the PHY.
Disable: RX lane is disabled
in the PHY.
Indicates if RX driver is
RX Status
enabled and serial data is
transmitted.
Enable: RX driver for the
RX Data enable Enable, Disable corresponding lane is
enabled.
Disable: RX driver for the
corresponding lane is
disabled.
continued...
104
UG-20237 | 2021.02.04
Indicates if RX (RX datapath,

RX settings) is in reset or
normal operating mode.
RX Reset Normal, Reset
Normal: RX is in normal
operating mode.
Reset: RX is in reset.
Indicates if the receiver has

lost the signal.
RX LOS <1,0> 1: Receiver loss of signal.
0: Receiver has a data
signal.
Indicates the CDR lock state.

CDR Lock Green, Red Green: CDR is locked.
Red: CDR is not locked.
RX CDR Indicates the CDR lock

mode.
Locked to Reference (LTR),
CDR Mode LTR: CDR is locked to
Locked to Data (LTD)
reference clock.
LTD: CDR is locked to data.
Gen3: Gen3 adaptation

mode. Indicates the RX adaptation
Adapt Mode
Gen4: Gen4 adaptation mode.
mode.
Indicates if the receiver is in

continuous adaptation.
Gen3: 1 • 0 - continuous adaptation
Adapt Continuous
Gen4: 1 off.
• 1 - continuous adaptation
on.
Gen3: 0 Indicates the RX equalization

RX ATT
Gen4: 0 attenuation level.
Gen3: 12 Indicates the RX CTLE boost

RX CTLE Boost
Gen4: 16 value.
Gen3: 2 Indicates the RX CTLE pole

RX CTLE Pole
Gen4: 2 value.
RX Equalization
Gen3: 5 Indicates the RX AFE first
RX VGA1
Gen4: 5 stage VGA gain value.
Gen3: 5 Indicates the RX AFE second

RX VGA2
Gen4: 5 stage VGA gain value.
Indicates the Receiver Figure

of Merit (FOM) / quality of
the received data eye. A
higher value indicates better
RX FOM <0-255> link equalization, with 8'd0
indicating the worst
equalization setting and
8'd255 indicating the best
equalization setting.
Indicates DFE adaptation is

enabled for taps 1 - 5.
DFE Enable Enable, Disable
Enable: DFE adaptation is
enabled for taps 1 - 5.
continued...
105
UG-20237 | 2021.02.04
Disable: DFE adaptation is

disabled for taps 1 - 5.
Indicates the adapted value

of DFE tap 1. This is a
DFE Tap1 adapted value <-128 to 127>
signed input (two's
complement encoded).

signed input (two's

signed input (two's

signed input (two's

signed input (two's
106
UG-20237 | 2021.02.04
Figure 42. Example of Receiver Settings
Eye Viewer
The P-Tile Debug Toolkit supports running eye tests for Intel devices with P-Tile. The
Eye Viewer tool allows you to set up and run eye tests, monitoring bit errors.
1. In the System Console Tools menu option, click on Eye View Tool.
107
UG-20237 | 2021.02.04
Figure 43. Opening the Eye Viewer
2. This will open a new tab Eye View Tool next to the Main View tab. Choose the
instance and channel for which you want to run the eye view tests.
Figure 44. Opening the Instance and Channel
3. Choose the eye vertical step setting from the drop-down menu. The eye view tool
allows you to choose between vertical step sizes of 1, 2, 4, 8.
Note: The time taken for the eye view tool to draw the eye varies with different
vertical step sizes (8 results in a faster eye plot when compared to 1).
Figure 45. Choosing the Step Size
4. The messages window displays information messages to indicate the eye view
tool's progress.
108
UG-20237 | 2021.02.04
Figure 46. Eye View Tool Messages
5. Once the eye plot is complete, the eye height, eye width and eye diagram are
displayed.
Figure 47. Sample Eye Plot
6.2.5. Enabling the P-Tile Link Inspector

To enable the Link Inspector, enable the option Enable Debug Toolkit in the PCIe
Configuration, Debug and Extension Options tab. The PCIe Link Inspector is
enabled by default if Enable Debug Toolkit is enabled.
Figure 48. Enabling the Link Inspector
109
UG-20237 | 2021.02.04
6.2.6. Using the P-Tile Link Inspector

The Link Inspector is found under the PCIe Link Inspector tab after opening the
Debug Toolkit:
Figure 49. View of the Link Inspector
When the Dump LTSSM Sequence to Text File button is initially clicked, a text file
(ltssm_sequence_dump_p*.txt) with the LTSSM information is created in the
location from where the System Console window is opened. Depending on the PCIe
topology, there can be up to four text files. Subsequent LTSSM sequence dumps will
append to the respective files.
110
UG-20237 | 2021.02.04
Figure 50. Example LTSSM Sequence Dump (Beginning)
111
UG-20237 | 2021.02.04
Figure 51. Example LTSSM Sequence Dump (End)
Each LTSSM monitor has a FIFO storing the time values and captured LTSSM states.
The FIFO is written when there is a state transition. When you want to dump the
LTSSM sequence, a single read of the FIFO status of the respective core is performed.
Depending on the empty status and how many entries are in the FIFO, successive
reads are executed.
112
UG-20237 | 2021.02.04
Send Feedback
7. Document Revision History
7.1. Document Revision History for the Intel FPGA P-Tile Avalon
Memory-mapped IP for PCI Express User Guide
Intel Quartus
Document Version IP Version Changes
Prime Version
Fixed the direction of the

2021.02.04 20.4 4.0.0 p0_pld_link_req_rst_o signal in the block
diagrams in the Avalon-MM Interface section.
Added parameters to enable the independent

resets for the x8x8 bifurcated mode to the
Parameters chapter.
Added a note to the Interface Clock Signals
2020.12.14 20.4 4.0.0 section to clarify that coreclkout_hip is an
internal clock only, and the Application layer must
use the p<n>_app_clk clock instead.
Replaced all references to coreclkout_hip with
p<n>_app_clk.
Removed the support for the Gen3 x4 256-bit and

Gen4 x4 256-bit configurations from the IP Core
2020.11.17 20.3 3.1.0 and Design Example Support Levels section. This
support may be available in a future release of
Intel Quartus Prime.
Updated the app_clk frequencies in the Clock

domains section.
2020.10.05 20.3 3.1.0
Added Root Port settings to the Avalon-MM
Settings section.
Added support for the Gen3 x8 Endpoint and

Gen4 x8 Endpoint modes to the Features chapter.
Updated the resource utilization numbers in the
2020.07.13 20.2 3.0.0
Resource Utilization chapter.
Added description for the Link Inspector in the
Debug Toolkit chapter.
Added the lane reversal and polarity inversion

support to the Features section.
2020.06.22 20.2 3.0.0 Updated the bit ranges for the Next Capability
Offset and Version fields in the Intel-Defined VSEC
Capability Registers section.
Added clarification that VCS is the only simulator

supported in the 20.1 release of Intel Quartus
2020.04.29 20.1 2.0.0
Prime. Also added that PIPE mode simulations are
not supported in this release.
continued...
7. Document Revision History
UG-20237 | 2021.02.04
Intel Quartus
Document Version IP Version Changes
Prime Version
Changed the operation mode names from DMA

Mode with Data Movers to Endpoint Mode with
Data Movers, and from Bursting Slave Mode to
Endpoint Mode.
Updated the document title to Intel FPGA P-Tile

Avalon memory mapped IP for PCI Express User
Guide to meet new legal naming guidelines.
2020.04.28 20.1 2.0.0 Updated the list of configurations supported in the
Features section.
Replaced the Configuration Slave Interface with
the Control Register Access Interface.
Added parameters in Intel Quartus Prime to

2019.12.16 19.4 1.1.0 control PASID and LTR.
Added MSI extended data support.
Added resource utilization numbers for the DMA

design example in Intel Stratix 10 DX devices.
2019.11.05 19.3 1.0.0 Added the step to choose Intel Stratix 10 DX
devices to the Generating the Design Example
section.
Removed a note containing a restriction on which

normal descriptors cannot be interrupted by
priority descriptors from the section Write Data
2019.10.28 19.3 1.0.0
Mover Avalon-ST Descriptor Sinks, because all
normal descriptors being processed cannot be
interrupted.
2019.10.23 19.3 1.0.0 Initial release.
114
UG-20237 | 2021.02.04
Send Feedback
A. Configuration Space Registers
A.1. Configuration Space Registers

In addition to accessing the Endpoint's configuration space registers by sending
Configuration Read/Write TLPs via the Avalon-ST interface, the application logic can
also gain read access to these registers via the Configuration Output Interface
(tl_cfg*). Furthermore, the Hard IP Reconfiguration Interface (a User Avalon-MM
interface) also provides read/write access to these registers.
For signal timings on the User Avalon-MM interface, refer to the Avalon Interface
Specifications document.
The table PCIe Configuration Space Registers describes the registers for each PF. To
calculate the address for a particular register in a particular PF, add the offset for that
PF from the table Configuration Space Offsets to the byte address for that register as
given in the table PCIe Configuration Space Registers.
Table 70. Configuration Space Offsets

Registers User Avalon-MM Offsets
Physical function 0 0x00000
Port Configuration and Status Register 0x104000
Debug (DBI) Register 0x104200, 0x104204
Table 71. PCIe Configuration Space Registers for x16/x8/x4 Controllers

Hard IP Configuration Space Corresponding Section in PCIe
Byte Address
Register Specification
x16 (Port 0) = 0x000 : 0x03C PCI Header Type 0/1 Configuration Type 0/1 Configuration Space Header
x8 (Port 1) = 0x000 : 0x03C Registers
x4 (Ports 2,3) = 0x000 : 0x03C
x16 (Port 0) = 0x040 : 0x044 Power Management PCI Power Management Capability
x8 (Port 1) = 0x040 : 0x044 Structure
x4 (Ports 2,3) = 0x040 : 0x044
continued...
UG-20237 | 2021.02.04

Byte Address
x16 (Port 0) = 0x050 : 0x064 MSI Capability MSI Capability Structure, see also PCI
x8 (Port 1) = 0x050 : 0x064 Local Bus Specification
x4 (Ports 2,3) = 0x050 : 0x064
x16 (Port 0) = 0x070 : 0x0A8 PCI Express Capability PCI Express Capability Structure
x8 (Port 1) = 0x070 : 0x0A8
x4 (Ports 2,3) = 0x070 : 0x0A8
x16 (Port 0) = 0x0B0 : 0x0B9 MSI-X Capability MSI-X Capability Structure, see also
x8 (Port 1) = 0x0B0 : 0x0B9 PCI Local Bus Specification
x4 (Ports 2,3) = 0x0B0 : 0x0B9
x16 (Port 0) = 0x0BC : 0x0FC Reserved N/A

x8 (Port 1) = 0x0BC : 0x0FC
x4 (Ports 2,3) = 0x0BC : 0x0FC
x16 (Port 0) = 0x100 : 0x144 Advanced Error Reporting (AER) Advanced Error Reporting Capability
x8 (Port 1) = 0x100 : 0x144 Structure
x4 (Ports 2,3) = 0x100 : 0x144
x16 (Port 0) = 0x148 : 0x164 Virtual Channel Capability Virtual Channel Capability Structure
x8 (Port 1) = 0x148 : 0x164
x4 (Ports 2,3) = 0x148 : 0x164
x16 (Port 0) = 0x178 : 0x17C Alternative Routing-ID Implementation ARI Capability Structure
x8 (Port 1) = 0x178 : 0x17C (ARI)
x4 (Ports 2,3) = N/A
x16 (Port 0) = 0x188 : 0x1B4 Secondary PCI Express Extended PCI Express Extended Capability
x8 (Port 1) = 0x188 : 0x1A4 Capability Header
x4 (Ports 2,3) = 0x188 : 0x1A4
x16 (Port 0) = 0x1B8 : 0x1E4 Physical Layer 16.0 GT/s Extended Physical Layer 16.0 GT/s Extended
x8 (Port 1) = 0x1A8 : 0x1CC Capability Capability Structure
x4 (Ports 2,3) = 0x1A8 : 0x1C8
x16 (Port 0) = 0x1E8 : 0x22C Margining Extended Capability Margining Extended Capability
x8 (Port 1) = 0x1D0 : 0x1F4 Structure
x4 (Ports 2,3) = 0x1CC : 0x1E0
x16 (Port 0) = 0x230 : 0x26C SR-IOV Capability SR-IOV Capability Structure

x8 (Port 1) = 0x1F8 : 0x234
x16 (Port 0) = 0x270 : 0x2F8 TLP Processing Hints (TPH) Capability TLP Processing Hints (TPH) Capability
x8 (Port 1) = 0x238 : 0x2C0 Structure
x4 (Ports 2,3) = 0x1E4 : 0x26C
x16 (Port 0) = 0x2FC : 0x300 Address Translation Services (ATS) Address Translation Services Extended
x8 (Port 1) = 0x2C4 : 0x2C8 Capability Capability (ATS) in Single Root I/O
Virtualization and Sharing Specification
x16 (Port 0) = 0x30C : 0x314 Access Control Services (ACS) Access Control Services (ACS)
x8 (Port 1) = 0x2D4 : 0x2DC Capability Capability
x4 (Ports 2,3) = 0x280 : 0x288
x16 (Port 0) = 0x318 : 0x324 Page Request Services (PRS) Capability Page Request Services (PRS) Capability
x8 (Port 1) = 0x2E0 : 0x2EC
continued...
116
UG-20237 | 2021.02.04

Byte Address
x16 (Port 0) = 0x328 : 0x32C Latency Tolerance Reporting (LTR) Latency Tolerance Reporting (LTR)
x8 (Port 1) = 0x2F0 : 0x2F4 Capability Capability
x16 (Port 0) = 0x330 : 0x334 Process Address Space (PASID) Process Address Space (PASID)
x8 (Port 1) = 0x2F8 : 0x2FC Capability Capability Structure
x16 (Port 0) = 0x338 : 0x434 RAS D.E.S. Capability (VSEC)

x8 (Port 1) = 0x300 : 0x3FC
x4 (Ports 2,3) = 0x2AC : 0x3A8
x16 (Port 0) = 0x470 : 0x478 Data Link Feature Extended Capability

x8 (Port 1) = 0x438 : 0x440
x4 (Ports 2,3) = 0x3E4 : 0x3EC
x16 (Port 0) = 0xD00 : 0xD58 Intel-defined VSEC

x8 (Port 1) = 0xD00 : 0xD58
x4 (Ports 2,3) = 0xD00 : 0xD58
A.1.1. Register Access Definitions

This document uses the following abbreviations when describing register accesses.
Table 72. Register Access Abbreviations

Abbreviation Meaning
RW Read and write access
RO Read only
WO Write only
RW1C Read write 1 to clear
RW1CS Read write 1 to clear sticky
RWS Read write sticky
Note: Sticky bits are not initialized or modified by hot reset.
A.1.2. PCIe Configuration Header Registers

The Corresponding Section in PCIe Specification column in the tables in the
Configuration Space Registers section lists the appropriate sections of the PCI Express
Base Specification that describe these registers.
117
UG-20237 | 2021.02.04
Figure 52. PCIe Type 0 Configuration Space Registers - Byte Address Offsets and Layout
31 24 23 16 15 87 0
0x000 Device ID Vendor ID
0x004 Status Command
0x008 Class Code Revision ID
0x00C 0x00 Header Type 0x00 Cache Line Size
0x010 BAR Registers
0x014 BAR Registers
0x018 BAR Registers
0x01C BAR Registers
0x020 BAR Registers
0x024 BAR Registers
0x028 Reserved
0x02C Subsystem Device ID Subsystem Vendor ID
0x030 Reserved
0x034 Reserved Capabilities Pointer
0x038 Reserved
0x03C 0x00 Interrupt Pin Interrupt Line
Figure 53. PCIe Type 1 Configuration Space Registers - Byte Address Offsets and Layout
31 24 23 16 15 87 0
0x0000 Device ID Vendor ID
0x004 Status Command
0x008 Class Code Revision ID
0x00C BIST Header Type Primary Latency Timer Cache Line Size
0x010 BAR Registers
0x014 BAR Registers
0x018 Secondary Latency Timer Subordinate Bus Number Secondary Bus Number Primary Bus Number
0x01C Secondary Status I/O Limit I/O Base
0x020 Memory Limit Memory Base
0x024 Prefetchable Memory Limit Prefetchable Memory Base
0x028 Prefetchable Base Upper 32 Bits
0x02C Prefetchable Limit Upper 32 Bits
0x030 I/O Limit Upper 16 Bits I/O Base Upper 16 Bits
0x034 Reserved Capabilities Pointer
0x038 Expansion ROM Base Address
0x03C Bridge Control Interrupt Pin Interrupt Line
Related Information
PCI Express Base Specification 4.0
118
UG-20237 | 2021.02.04
A.1.3. PCI Express Capability Structures

The layouts of the most basic Capability Structures are provided below. Refer to the
PCI Express Base Specification for more information about these registers.
Figure 54. Power Management Capability Structure - Byte Address Offsets and Layout
31 24 23 16 15 87 0
0x040 Capabilities Register Next Cap Ptr Capability ID
PM Control/Status
0x04C Data Power Management Status and Control
Bridge Extensions
Figure 55. MSI Capability Structure
31 24 23 16 15 87 0
Message Control
0x050 Configuration MSI Control Status Next Cap Ptr Capability ID
Register Field Descriptions
0x054 Message Address
0x058 Message Upper Address
0x05C Reserved Message Data
Figure 56. PCI Express Capability Structure - Byte Address Offsets and Layout
In the following table showing the PCI Express Capability Structure, registers that are not applicable to a
device are reserved.
31 24 23 16 15 87 0
PCI Express
0x070 PCI Express Capabilities Register Next Cap Pointer
Capabilities ID
0x074 Device Capabilities
0x078 Device Status Device Control
0x07C Link Capabilities
0x080 Link Status Link Control
0x084 Slot Capabilities
0x088 Slot Status Slot Control
0x08C Root Capabilities Root Control
0x090 Root Status
0x094 Device Compatibilities 2
0x098 Device Status 2 Device Control 2
0x09C Link Capabilities 2
0x0A0 Link Status 2 Link Control 2
0x0A4 Slot Capabilities 2
0x0A8 Slot Status 2 Slot Control 2
119
UG-20237 | 2021.02.04
Figure 57. MSI-X Capability Structure

31 24 23 16 15 87 3 2 0
0x0B0 Message Control Next Cap Ptr Capability ID
MSI-X
0x0B4 MSI-X Table Offset Table BAR
Indicator
MSI-X
Pending
0x0B8 MSI-X Pending Bit Array (PBA) Offset Bit Array
- BAR
Indicator
Figure 58. PCI Express AER Extended Capability Structure

31 16 15 0
0x100 PCI Express Enhanced Capability Register
0x104 Uncorrectable Error Status Register
0x108 Uncorrectable Error Mask Register
0x10C Uncorrectable Error Severity Register
0x110 Correctable Error Status Register
0x114 Correctable Error Mask Register
0x118 Advanced Error Capabilities and Control Register
0x11C Header Log Register
0x12C Root Error Command Register
0x130 Root Error Status Register
0x134 Error Source Identification Register Correctable Error Source Identification Register
Related Information
PCI Express Base Specification 4.0
120
UG-20237 | 2021.02.04
A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure

Figure 59. Physical Layer 16.0 GT/s Extended Capability Structure
A.1.5. MSI-X Registers

This section describes the registers previously shown in the MSI-X capability structure.
Table 73. MSI-X Control Register

Bit Location Description Access Default Value
31 MSI-X Enable: This bit must RW 0

be set to enable the MSI-X
interrupt generation.
30 MSI-X Function Mask: This RW 0

bit can be set to mask all
MSI-X interrupts from this
function.
29:27 Reserved RO 0
26:16 Size of the MSI-X table RO Programmed via the

(number of MSI-X interrupt programming interface.
vectors). The value in this
field is one less than the size
of the table set up for this
function. Maximum value is
0x7FF (2048 interrupt
vectors).
This field is shared among
all VFs attached to one PF.
15:8 Next Capability Pointer RO Programmed via the

Points to the PCI Express programming interface.
Capability.
7:0 Capability ID assigned by RO 0x11

PCI-SIG.
121
UG-20237 | 2021.02.04
Table 74. MSI-X Table Offset Register

2:0 BAR Indicator Register: RO Programmed via the

Specifies the BAR programming interface.
corresponding to the
memory address range
where the MSI-X table of
this function is located (000
= VF BAR0, 001 = VF BAR1,
…, 101 = VF BAR5).
31:3 Offset of the memory RO Programmed via the

address where the MSI-X programming interface.
table is located, relative to
the specified BAR. The
address is extended by
appending three zeroes to
make it Qword aligned.
Table 75. MSI-X Pending Bit Array Register

2:0 BAR Indicator Register: RO Programmed via the

Specifies the BAR programming interface.
corresponding to the
memory address range
where the Pending Bit Array
of this function is located
(000 = VF BAR0, 001 = VF
BAR1, …, 101 = VF BAR5).
31:3 Offset of the memory RO Programmed via the

address where the Pending programming interface.
Bit Array is located, relative
to the specified BAR. The
address is extended by
appending three zeroes to
make it Qword aligned.
A.2. Intel-Defined VSEC Capability Registers

Table 76. Intel-Defined VSEC Capability Registers (0xD00 : 0xD58)
31 : 20 19 : 16 15 : 8 7:0 PCIe Byte Offset
Next Cap Offset Version PCI Express Extended Capability ID 00h
VSEC Length VSEC Rev VSEC ID 04h
Intel Marker 08h
JTAG Silicon ID DW0 0Ch
JTAG Silicon ID DW1 10h

continued...
122
UG-20237 | 2021.02.04
31 : 20 19 : 16 15 : 8 7:0 PCIe Byte Offset
CvP Status User Configurable Device/Board ID 1Ch
CvP Mode Control 20h
CvP Data 2 24h
CvP Data 28h
CvP Programming Control 2Ch
General Purpose Control and Status 30h
Uncorrectable Internal Error Status Register 34h
Uncorrectable Internal Error Mask Register 38h
Correctable Error Status Register 3Ch
Correctable Error Mask Register 40h
SSM IRQ Request & Status 44h
SSM IRQ Result Code 1 Shadow 48h
SSM IRQ Result Code 2 Shadow 4Ch
SSM Mailbox 50h
SSM Credit 0 Shadow 54h
SSM Credit 1 Shadow 58h
A.2.1. Intel-Defined VSEC Capability Header (Offset 00h)

Table 77. Intel-Defined VSEC Capability Header
Bits Register Description Default Value Access
[31:20] Next Capability Pointer. Variable RO

Value is the starting address
of the next Capability
Structure implemented, if
any. Otherwise, NULL. Refer
to the Configuration Address
Map.
[19:16] Capability Version. PCIe 0x1 RO

specification-defined value
for VSEC Capability Version.
[15:0] Extended Capability ID. PCIe 0x000B RO

specification-defined value
for VSEC Extended
Capability ID.
123
UG-20237 | 2021.02.04
A.2.2. Intel-Defined Vendor Specific Header (Offset 04h)

Table 78. Intel-Defined Vendor Specific Header
[31:20] VSEC Length. Total length of 0x5C RO

this structure in bytes.
[19:16] VSEC Rev. User configurable k_vsec_rev_i RO

VSEC revision.
[15:0] VSEC ID. User configurable 0x1172 RO

VSEC ID. The default value
is 0x1172 (the Intel Vendor
ID), but you can change this
ID to your own Vendor ID.
A.2.3. Intel Marker (Offset 08h)

Table 79. Intel Marker
Intel Marker - An additional

marker for standard Intel
[31:0] programming software to be 0x41721172 RO
able to verify that this is the
right structure.
A.2.4. JTAG Silicon ID (Offset 0x0C - 0x18)

This read-only register returns the JTAG Silicon ID. Intel programming software uses
this JTAG ID to ensure that is is using the correct SRM Object File (*.sof).
These registers are only good for Port 0 (PCIe Gen4 x16). They are blocked for the
other Ports.
Table 80. JTAG Silicon ID Registers

Bits Register Description Default Value(6) Access
[127:96] JTAG Silicon ID DW3 Unique ID RO
A.2.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D)

This register provides a user configurable device or board ID so that the user software
can determine which .sof file to load into the device.
This register is only available for Port 0 (PCIe Gen4 x16). It is blocked for the other
Ports.
(6) Because the Silicon ID is a unique value, it does not have a global default value.
124
UG-20237 | 2021.02.04
Table 81. User Configurable Device and Board ID Register

[15:0] This register allows you to From configuration bits RO

specify the ID of the .sof
file to be loaded.
A.2.6. General Purpose Control and Status Register (Offset 0x30)

This register provides up to eight I/O pins each for Application Layer Control and
Status requirements. This feature supports Partial Reconfiguration of the FPGA fabric.
Partial Reconfiguration only requires one input pin and one output pin. The other
seven I/Os make this interface extensible.
Table 82. General Purpose Control and Status Register

[31:16] Reserved. N/A RO
[15:8] General Purpose Status. The 0x00 RO

Application Layer can read
these status bits. These bits
are only available for Port 0
(PCIe Gen4 x16). They are
blocked for the other Ports.
[7:0] General Purpose Control. 0x00 RW

The Application Layer can
write these control bits.
These bits are only available
for Port 0 (PCIe Gen4 x16).
They are blocked for the
other Ports.
A.2.7. Uncorrectable Internal Error Status Register (Offset 0x34)

This register reports the status of the internally checked errors that are uncorrectable.
When these specific errors are enabled by the Uncorrectable Internal Error
Mask register, they are forwarded as Uncorrectable Internal Errors.
Note: This register is for debug only. Only use this register to observe behavior, not to drive
logic custom logic.
Table 83. Uncorrectable Internal Error Status Register

[31:13] Reserved 0x0 RO
[12] Debug Bus Interface (DBI) 0x0 RW1CS

access error status from
Config RAM block.
[11] Uncorrectable ECC error 0x0 RW1C

from Config RAM block.
[8] RX Transaction Layer parity 0x0 RW1CS

error reported by the IP
core.
continued...
125
UG-20237 | 2021.02.04
[7] TX Transaction Layer parity 0x0 RW1CS

error reported by the IP
core.
[6] Uncorrectable Internal Error 0x0 RW1CS

reported by the FPGA.
[5] cvp_config_error_latch 0x0 RW1CS

ed: Configuration error
detected in CvP mode is
reported as an uncorrectable
error. Set whenever
ssm_cvp_config_error of
the SSM Scratch CvP Status
register bit[1] rises in CvP
mode. This bit is only
available for Port 0 (PCIe
Gen4 x16), but not for the
other Ports.
Note: The access code RW1CS represents Read Write 1 to Clear Sticky.
A.2.8. Uncorrectable Internal Error Mask Register (Offset 0x38)

This register controls which errors are forwarded as internal uncorrectable errors.
Table 84. Uncorrectable Internal Error Mask Register

[12] Mask for Debug Bus 0x1 RWS

Interface (DBI) access error.
[11] Mask for Uncorrectable ECC 0x1 RWS

error from Config RAM block.
[8] Mask for RX Transaction 0x1 RWS

Layer parity error reported
by the IP core.
[7] Mask for TX Transaction 0x1 RWS

Layer parity error reported
by the IP core.
[6] Mask for Uncorrectable 0x1 RWS

Internal error reported by
the FPGA.
[5] Mask for Configuration Error 0x0 RWS

detected in CvP mode. This
bit is only available for Port
0 (PCIe Gen4 x16), but not
for the other Ports.
Note: The access code RWS stands for Read Write Sticky, meaning that the value is retained
after a soft reset of the IP core.
126
UG-20237 | 2021.02.04
A.2.9. Correctable Internal Error Status Register (Offset 0x3C)

The Correctable Internal Error Status register reports the status of the
internally checked errors that are correctable. When these specific errors are enabled
by the Correctable Internal Error Mask register, they are forwarded as
correctable internal errors. This register is for debug only. Only use this register to
observe behavior, not to drive custom logic
Table 85. Correctable Internal Error Status Register

[11] Correctable ECC error status 0x0 RW1CS

from Config RAM.
[6] Correctable Internal Error 0x0 RW1CS

reported by the FPGA.
[5] cvp_config_error_latch 0x0 RW1CS

ed: Configuration error
detected in CvP mode (to be
reported as correctable) -
Set whenever
cvp_config_error rises
while in CvP mode. This bit
is only available for Port 0
(PCIe Gen4 x16), but not for
the other Ports.
A.2.10. Correctable Internal Error Mask Register (Offset 0x40)

This register controls which errors are forwarded as internal correctable errors.
Table 86. Correctable Internal Error Mask Register

[11] Mask for Correctable ECC 0x1 RWS

error status for Config RAM.
[10:7] Reserved 0x0 RWS
[6] Mask for Correctable 0x1 RWS

Internal Error reported by
the FPGA.
[5] Mask for Configuration Error 0x1 RWS

detected in CvP mode. This
bit is only available for Port
0 (PCIe Gen4 x16), but not
for the other Ports.
[4] Reserved 0x1 RWS
[3:0] Reserved 0x0 RWS
127

P-Tile Avalon Memory-Mapped IP For PCI Express User Guide

Uploaded by

Copyright:

Available Formats

You might also like

P-Tile Avalon Memory-Mapped IP For PCI Express User Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

P-Tile Avalon Memory-Mapped IP For PCI Express User Guide

Uploaded by

Copyright:

Available Formats

Intel® FPGA P-Tile Avalon® Memory-

mapped IP for PCI Express* User

Subscribe UG-20237 | 2021.02.04

4.10.2. Configuration Registers Access................................................................. 79

The P-Tile Avalon memory mapped IP for PCIe consists of:

Endpoint (EP) Yes Yes N/A

Root Port (RP) (1) N/A Yes

Note: Gen1/Gen2 configurations are supported via link down-training.

Table 2. Outstanding Non-Posted Requests Supported

0 x16 64, 512 (*)

Note: (*) : 512 outstanding Non-Posted requests support may be available in a

1.3. Release Information

Intel Quartus Prime Version 20.4

Release Date December 2020

Ordering Codes No ordering code is required

1.4. Device Family Support

Table 4. Device Family Support

Intel Stratix® 10 DX Preliminary support

Intel Agilex™ Preliminary support

1.5. Performance and Resource Utilization

200 MHz (Intel Stratix

200 MHz (Intel Stratix

350 MHz (Intel Stratix

x4 256-bit 125 MHz -1, -2

Gen3 x8 512-bit 125 MHz -1, -2

x16 512-bit 250 MHz -1, -2

Table 6. Resource Utilization of the Avalon-MM IP for PCI Express IP Core

DMA Gen3 x16, EP Intel Agilex 17116 120 42940

DMA Gen4 x16, EP Intel Agilex 16963 120 45425

DMA Gen4 x8, EP Intel Agilex 16275 97 41025

1.6. IP Core and Design Example Support Levels

PCIe IP Support Design Example Support

Gen4 x16 512-bit SCTH (††) S C T H (†) (††)

Gen4 x8/x8 512-bit SCTH N/A S C T H (†) N/A

Gen3 x16 512-bit SCTH (††) S C T H (†) (††)

Gen3 x8/x8 512-bit SCTH N/A S C T H (†) N/A

PCIe IP Support Design Example Support

Gen4 x16 512-bit SCTH (††) S C T H (†) (††)

Gen4 x8/x8 512-bit SCTH N/A S C T H (†) N/A

Gen3 x16 512-bit SCTH (††) S C T H (†) (††)

Gen3 x8/x8 512-bit SCTH N/A S C T H (†) N/A

2. IP Architecture and Functional Description

2.1. Top-Level Architecture

Figure 1. P-tile Avalon-MM IP for PCI Express top-level block diagram

P-Tile EMIB FPGA Fabric

PLL A/B (MAC)

Configuration Mode 0 Gen3x16 or Gen4x16 EP/RP x16

Configuration Mode 1 Gen3x8/Gen3x8 or Gen4x8/Gen4x8 EP x16, x8

2.1.1. Avalon-MM Bridge Architecture

Table 10. Operating Modes of the Avalon-MM Bridge

Bursting Avalon-MM Bursting Avalon-MM

Figure 2. P-Tile Avalon-MM Bridge Block Diagram

Width and Rate Adaptor

P-Tile PCle Avalon-ST

P-Tile PCle Hard IP

2.1.2. Clock Domains

Figure 3. Clock Domains

FPGA Fabric P-Tile x16 core_clk