Pensando SSDK SDN Policy Offload User Guide

SDN Policy Offload
Reference Pipeline
User Guide
AMD Pensando Software-in-Silicon Development Kit
UG1668 (v1.0) November 30, 2023
AMD Adaptive Computing is creating an environment where

employees, customers, and partners feel welcome and included. To
that end, we’re removing non-inclusive language from our products
and related collateral. We’ve launched an internal initiative to remove
language that could exclude people or reinforce historical biases,
including terms embedded in our software and IPs. You may still find
examples of non-inclusive language in our older products as we work
to make these changes and align with evolving industry standards.
Follow this link for more information.
Table of Contents
Chapter 1: Introduction.............................................................................................. 3
SSDK SDN Policy Offload Pipeline Overview............................................................................ 4
Ready-to-Deploy Libraries..........................................................................................................5
Build Your Own Libraries............................................................................................................6
Chapter 2: Architecture...............................................................................................9
Overview.......................................................................................................................................9
SDN Policy Implementation..................................................................................................... 10
Topology for Testing SDN Policy Offload Reference Pipeline..............................................12
Chapter 3: Building and Running the Reference Pipeline..................... 15

About the SSDK..........................................................................................................................15
Building a Simulator Image..................................................................................................... 16
Managing the Simulation Environment................................................................................. 17
Building a DSC Firmware Image..............................................................................................18
Copying an Image to the DSC..................................................................................................18
Loading an Image..................................................................................................................... 23
Starting the DP-App.................................................................................................................. 24
Cleaning the Build..................................................................................................................... 25
Performing a Packet Test......................................................................................................... 25
Troubleshooting........................................................................................................................ 25
Appendix A: Additional Resources and Legal Notices............................. 26

Finding Additional Documentation......................................................................................... 26
Support Resources.................................................................................................................... 26
References..................................................................................................................................26
Revision History......................................................................................................................... 26
Please Read: Important Legal Notices................................................................................... 27
UG1668 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK SDN Policy Offload Reference Pipeline User Guide 2
Chapter 1: Introduction
Chapter 1
Introduction
The AMD Pensando™ Software-in-Silicon Development Kit (SSDK) allows developers to port and
create custom implementations of networking and storage functions or services for the AMD
Pensando DPU.
This document provides an overview of the Software-Defined Networking (SDN) Policy Offload
reference pipeline and details the P4 program including a set of ready to deploy precompiled
libraries and ready to use code for RxDMA/TxDMA/SxDMA pipelines for routing, policy
evaluation, and flow aging. The P4 pipeline programmability provides flexible software-defined
constructs that allow you to:
• Quickly develop your own networking software

• Load that to an AMD Pensando DPU
• Test
• Deploy
The reference pipeline supplies P416 source code for P4I and P4E modules for protocol
processing and P416 based libraries in binary format for P4 RxDMA and P4 TxDMA that you can
access via APIs for handling message transfer with host and local CPU. This combination of
source code and libraries facilitates the implementation of the reference pipeline. The P4 binary
can be deployed along with your P4 code on the same DPU to integrate its functionality into
your system. This approach streamlines the implementation process and ensures seamless
compatibility.
The AMD Pensando DPU can offload and accelerate a wide range of SDN, security, and storage
features and capabilities. The SSDK makes it easy to develop and deploy solutions using its
flexible P4 programmable data path and rich APIs.
The traditional approach to selecting ASICs or CPUs for switches, security appliances and routers
is to consider the cost, table size, flexibility, and performance of each option. High-end ASICs are
the most expensive option, but they offer the best performance and can support large-scale
routing tables. Lower-end ASICs are less expensive but they may be constrained by routing table
size, buffer size and performance. ASICs are efficient at performing specific tasks it was designed
for but lack flexibility to add new features or services without a respin. CPUs are the least
expensive option, but they offer the worst performance and can only support small routing
tables, limited session scale and low connections per second scale, but offer flexibility as it
relates to adding new features in software.
UG1668 (v1.0) November 30, 2023

Send Feedback
An AMD Pensando DPU with a programmable P4 pipeline is a software-defined data plane that
can be programmed to perform various data plane functions in hardware to meet high-
performance and scale requirements. This means that new functions and services can be added
without having to redesign the hardware, which can save time and resources; this lets you keep
up with the pace of rapid new requirements to meet your product requirements without
sacrificing performance and scale.
Benefits of using a software-defined data plane include:
• Flexibility: Software-defined data planes are highly programmable and can support new
functions and services with performance and scale. This makes them ideal for data centers
that need to adapt to change and require high performance.
• Agility: Software-defined data planes can be programmed quickly and easily, without having
to go through a lengthy design and manufacturing process. This makes it possible to quickly
deploy new features and services.
• Cost savings: Software-defined data planes can help to reduce costs by eliminating the need
to respin hardware when new functions are required.
• Performance: The AMD PensandoPensando DPU is purpose-built to offload and accelerate

infrastructure services from the CPU.
SSDK SDN Policy Offload Pipeline Overview

The SDN Policy Offload reference pipeline provides an example of how to use an AMD-
developed, ready-to-deploy P4 library to build a high-performance, highly scalable data path
application. The P4 library implements the routing table and ACL security lookup in P4-DMA,
achieving much higher performance compared to performing the legacy approach where the
routing table lookup and ACL lookup take place on CPU cores. By storing the routing tables and
ACL table on the DDR memory, the solution can scale to millions, or tens of millions of entries
without the high cost of HBM or TCAM used in ASIC designs. Connection tracking and flow
aging are handled in the P4 pipeline, further offloading the CPU unlike legacy designs. With the
routing table lookup and ACL lookup offloaded and accelerated by P4-DMAs and connection
tracking offloaded by P4, the P4 library allows you to build a stateful services application with
very high CPS or stateless services applications (not flow-aware or connection session aware)
such as cost-effective router with large scale routing table and ACL. You can do various
manipulations on the packet in the dataplane, but before the flow is installed, the packet must go
through DMA for the route lookup and policy evaluation and with the option to even enforce all
packets of a flow to go through route and policy evaluation.
UG1668 (v1.0) November 30, 2023

Send Feedback
Ready-to-Deploy Libraries
The SDN Policy Offload reference pipeline includes a robust list of functions and services to
jumpstart your development and accelerate your path to production deployments by including
ready to deploy libraries. The following is a list of services ready-to-deploy. You can move to
production with the included ready to deploy libraries and customize the other stages in the
pipeline.
• Routing (LPM and Priority-Based Route Lookup).

○ Current configuration can support up to 4 million routes. Validation has been carried out
for one million routes at present.
○ Routing using longest prefix matching (LPM) is implemented as a P4 program running on
the P4-DMA pipeline of the DPU. Libraries and binaries required for this routing
functionality are packaged as part of the SSDK tarball.
○ Route lookups in the P4DMA data path.
○ APIs are exposed to programmers to use route-table config objects and APIs to initialize,
create, delete, and update.
○ Multi-tenancy: A Table ID is added to the LPM routing table implementation allowing the
lookup to be achieved by the key: table-id and DIP prefix/SIP prefix. The Table ID in this
case can be interpreted as the VRF. You can define the Table ID. An example is using the
VID in the VxLAN header as the Table ID.
○ Supports both IPv4 and IPv6 based route tables.
• SDN/Security Policy
○ 1K Rules and 256K source IP prefixes. 256K destination IP prefixes per policy supported
with reference pipeline.
○ Rule priority is supported.
○ Bi-directional policy enforcement (Ingress and Egress). A unique policy can be applied in
each direction.
○ Incremental rule add/delete.
○ Policy lookups in the P4-DMA data path.
○ APIs are exposed to programmers to use route-table config objects and APIs to initialize,
create, delete, and update.
○ Supports both IPv4 and IPv6 based policies.
○ Flow logging is not included in the reference pipeline but can be added.
UG1668 (v1.0) November 30, 2023

Send Feedback
• Hardware Assisted Flow Aging

○ Performed as a hardware assist for flow aging where the P4 pipeline can notify the dp-app
or any control plane application using the well defined APIs of the expired sessions so that
the application can take appropriate actions to delete the corresponding flow entries from
the P4 flow table.
○ Scanners: P4+ programs that scan the entire depth of a session table to detect expired
flows.
○ Pollers: Pollers are specific threads on the dp-app that poll a dedicated poller queue. A
poller queue is a queue that receives expiry map bitmaps from a scanner program.
○ Idle Aging: Currently, two types of inactivity timeout configurations are supported:
stateless and stateful.
○ Connection Track Aging is implemented for TCP connections only. P4 pipeline tracks TCP
state and per flow TCP state is maintained in session tracking table along with last seen
sequence and ack numbers. TCP state transitions tracked are:
- INIT
- SYN_SENT
- ESTABLISHED (post 3 way handshake)
- FIN1_RECEIVED
- FIN2_RECEIVED
- RST
- CLOSED
○ Closed Session (A/B copy): Once the connection track program detects that the connection
is closed, the closed session bitmaps are managed as A and B copy for synchronizing the
update between P4 and P4+ programs.
○ Show/Debug commands.
○ Testing Tool: Flow Creation: send a packet with a SCAPI script.
Build Your Own Libraries

Besides the ready-to-deploy P4 code for the P4I pipelines, P4E pipelines, binaries, and libraries
for RxDMA/TxDMA/SxDMA, you can build, enhance, or combine with your own P4 code. The
following is a list of such libraries:
• Flow offload (64 million sessions) with connection tracking.

• VLAN and VXLAN Encap/DeCap.
• Route Guard: A packet can be dropped without using flow and policy rules.
UG1668 (v1.0) November 30, 2023

Send Feedback
• NAT: Inner IP header (Source IP, Destination IP, Source port and Destination Port).
The above list gets you started and below is a list of SDN services that can be implemented using
the SSDK on an AMD Pensando DPU:
• Virtual Networking (Overlay)

○ Virtual IP Spaces.
○ Tenant aware routing (VRFs).
○ Network address translation for inter-VPC or out-bound traffic.
○ Capable of supporting:
- Static 1:1 NAT
- Many to 1
- Source
- Destination
- Twice NAT
○ Ingress and egress NAT policy direction.
○ Virtual network subnets: User defined subnets, custom IPAM l2/l3 forwarding with default
gateway.
○ Network services: NACLs, firewall, security groups, load balancing, encryption, ERSPAN
(vTAP), QoS etc.
○ VPN services: IPsec and PSP encryption services.
○ Metering
• Traffic Engineering
○ Segment routing: SRv6 and SR-MPLS.
○ Network service mesh: Policy driven service functions chaining.
○ Custom routing via workload APIs or via BGP.
○ Overlay mapping tables.
○ Local mapping.
○ Remote mapping.
○ Source RPF security mapping (SIP to TEP mapping).
○ Routing tables.
○ Stateless network ACL tables.
UG1668 (v1.0) November 30, 2023

Send Feedback
• Physical Network (Underlay) Integration

○ BGP based host routed network.
○ LLDP for topology awareness.
○ LACP for bonding up L2 to TORs.
○ Topology Map: Provides the workloads with TOR connectivity.
Contact your AMD sales team to discuss SSDK enhancement or custom solution requests.
UG1668 (v1.0) November 30, 2023

Send Feedback
Chapter 2: Architecture
Chapter 2
Architecture
The SDN Policy Offload reference pipeline implements a set of SDN policy and networking
services, ready to deploy. Similar to the other reference pipelines, it provides insight into P4 and
its capabilities to develop your own services. Despite its relatively simple structure, the SDN
reference pipeline showcases the power of P4, that can be used for building custom networking
and security services such as IPsec, or any other stateless or stateful networking services without
compromising the performance of the networking device the DPU resides in.
Overview
The SDN Policy Offload pipeline is a bump-in-the-wire (BITW) implementation ideal for a
SmartSwitch or an appliance form factor as shown in Figure 1: BITW Implementation for
SmartSwitch Appliances. When deployed inside a SmartSwitch as a top-of-rack device or in an
appliance, the DPU is not visible externally because it is integrated into the device providing
network service acceleration to any traffic ingressing and egressing the device. When the DPU is
deployed in an appliance for service aggregation in the data center or a colocation facility, it
connects to the network devices and performs network and security services for various
workloads and network traffic with the help of traffic redirection to the service appliance.
The DPU connects to a switching ASIC as a BITW to control packets to and from the wire and
send it back to the switching ASIC forwarded to its next hop or destination. A single DPU can
provide multiple networking services in parallel, but also multiple DPUs can be connected to
scale out performance or achieve high availability. The BITW approach includes an appliance
mode using an x86 system with DPUs deployed in the PCIe® slots.
UG1668 (v1.0) November 30, 2023

Send Feedback
Figure 1: BITW Implementation for SmartSwitch Appliances
DPU ×N x86 System

Optional
200G TX 200G TX
Control Plane
200G RX 200G RX
DPU ×N
ASIC
Traffic Traffic
To/From To/From
Network Network
X28791-110623
The SDN Policy Offload pipeline can be easily enhanced to support a host-to-network
deployment with additional services, as shown in Figure 2: Host to Network Implementation.
Figure 2: Host to Network Implementation
x86 System pf0

Traffic
To/From Host
DPU
Traffic
To/From
Network
X28792-110623
SDN Policy Implementation

The SDN Policy Offload Pipeline provides a ready-to-deploy P4 library to accelerate:
• SDN policy evaluation (LPM table lookup for forwarding decision)

• Metering
• ACL lookup for security
• Flow session aging without consuming CPU cycles
The P4 tables can be programmed using the generated C++ APIs that support Create, Read,
Update, Delete (CRUD) operations. Figure 3: SDN Policy Offload API Model shows the API
model.
UG1668 (v1.0) November 30, 2023

Send Feedback
Figure 3: SDN Policy Offload API Model
API API
AMD Library in P4 AMD Library in P4

TxDMA RxDMA
Packet Buffer
Traffic Manager
Custom P4 Code Custom P4 Code

P4 Egress P4 Ingress
Eth 1/1 Eth 1/2
X28809-110823
The ready to deploy services can be deployed along with custom-developed P4 code on the
same DPU to achieve high performance to support very high session setup rate, high scale CPS
and large scale flow table with hardware-based flow aging. The P4 programs can be used to
enhance or design and develop stateless functions (not flow aware or session aware) or stateful
service by combining the AMD developed P4 libraries with the flow and session table placed in
P4I and P4E pipelines.
Routing and policies are implemented in P4 RxDMA and P4 TxDMA that are exercised for all flow
miss packets. A typical flow miss packet path is as follows:
Uplink Port → Ingress (flow-miss) → RxDMA → TxDMA → Egress → RxDMA → DP App →

TxDMA → Ingress → Egress → Uplink Port
On receiving a flow-miss packet containing metadata about LPM and security policy results that
the P4 accelerates, the Dp-App installs the flow with the corresponding policy and routing
information, and reinjects the packet into the P4 Ingress pipeline. The next packets for the flow
take the following flow-hit path and get forwarded in P4:
Uplink Port → Ingress (flow-hit) → Egress → Uplink Port
Figure 4: Packet Flow-Miss and Flow-Hit Paths shows the packet flow-miss and flow-hit paths.
UG1668 (v1.0) November 30, 2023

Send Feedback
Figure 4: Packet Flow-Miss and Flow-Hit Paths
Polls on Datapath LIF
DP-App
NOC
LPM Based Routing LPM Based Routing
P4 TxDMA P4 RxDMA
Packet Buffer
RxDMA Traffic Manager
TxDMA
Closed
Session Tunnel LIF
Sessions
FLOW NACL Stats
Session KEY
Rewrite Next Hop
Track
P4 Egress P4 Ingress
Idle Aging + Connection Tracking
Eth 1/1 Eth 1/2 SxDMA
X28811-110823
Topology for Testing SDN Policy Offload

Reference Pipeline
This section describes two topologies for testing the architecture.
Single DSC Topology
Figure 5: Single DSC Topology for Testing shows a single Distributed Services Card (DSC)
topology used for testing the architecture. The VXLAN packets are sent from workload-1 (WL-1),
which creates an initiator flow (iflow) for the inner IP and a responder flow (rflow) for the reverse
path from workload-2 (WL-2). An unencapsulated IP packet is returned so that the existing rflow
is hit and the packet is forwarded or dropped based on policy and route evaluation.
UG1668 (v1.0) November 30, 2023

Send Feedback
Figure 5: Single DSC Topology for Testing
Elba DSC-1
AD-App SDN Policy Acceleration Reference

Arm P4-Pipeline
Eth 1/1 Eth 1/2
VXLAN Encapsulated IP Packet
WL-1 WL-2
X28814-110823
Dual DSC Topology
Figure 6: Dual DSC Topology for Testing shows a dual DSC topology. Two DSCs are connected
back-to-back using port Eth 1/2. The other uplink ports of DSC1 and DSC2 (Eth1/1) are
connected to Ixia traffic generator ports 1 and 2, respectively. The SDN Policy Offload reference
P4 program runs on both DSCs.
DSC-1 is based on a permitted security policy and route and route LPM lookup routes IP traffic
from port-1 to DSC-2 via port Eth1/2.
DSC-2 is based on a permitted security policy and LPM route evaluation forwards the IP packet
to Ixia port-2. For performance analysis, traffic should be started and run symmetrically from Ixia
port-1 and port-2 for testing bi-directional throughput.
UG1668 (v1.0) November 30, 2023

Send Feedback
Figure 6: Dual DSC Topology for Testing
Traffic Generator (IXIA)
IP Packet
Eth 1/1
Elba DSC-1
Dp-App SDN Policy Acceleration Reference

Arm P4-Pipeline
Eth 1/2
VXLAN
Encapsulated
Eth 1/2
Elba DSC-2
Dp-App SDN Policy Acceleration Reference

Arm P4-Pipeline
Eth 1/1
IP Packet
Traffic Generator (IXIA)

X28827-110923
UG1668 (v1.0) November 30, 2023

Send Feedback
Chapter 3: Building and Running the Reference Pipeline
Chapter 3
Building and Running the Reference

Pipeline
This chapter describes how to build and run the reference pipeline using the Software-in-Silicon
Development Kit (SSDK).
Note: This chapter assumes you already have installed the developer environment. If not refer to the
following:
• The SSDK Quick Start Guide or Getting Started Guide for installation of the SSDK.
• The SSDK Getting Started Guide to setup your environment.
About the SSDK

The SSDK is installed as a container. It is delivered in three separate software packages:
• A container image that constitutes the build environment and includes generic tools, such as
the Linaro Arm® cross-compile toolchain.
• A tarball with the source code of the reference pipelines and AMD-specific tools, such as the
P4 compiler and the Distributed Services Card (DSC) simulator.
• A tarball with documentation.
The following table lists installed directories and their contents.
Table 1: SSDK Directories
Directory Contents
/sw/nic SSDK main folder for build scripts
/sw/nic/rudra Relevant code and files for development
/sw/nic/rudra/src/pds-core Core Application, Environment Initialization, Starts services,
PDS Agent, gRPC Listener, PDS Layer API, and Common
Libraries
/sw/nic/rudra/src/lib Common code and files for all pipelines, LIFS, Memory Utils,
Debug, Tracing, and Interface Management
/sw/nic/rudra/src/<reference-pipeline> The reference pipelines
UG1668 (v1.0) November 30, 2023

Send Feedback
Table 1: SSDK Directories (cont'd)
Directory Contents
/sw/nic/rudra/src/<reference-pipeline>/p4/ P4 Source Code for the specific pipeline
p4-16/
/sw/nic/rudra/src/conf Platform, Pipeline, and Infrastructure configuration files
/sw/nic/rudra/src/<reference-pipeline>/dp Dataplane and Pipeline specific DP application
/sw/nic/tools The developer tools
Note: This container can be used to build both x86 and Arm images.
Building a Simulator Image

To build a simulator image, do the following within the container:
1. Change directory to the main folder for build scripts:

cd /sw/nic
2. Clean the build directory:

ARCH=x86_64 ./tools/build.sh --p4_program sdn_policy_offload --clean
3. Build the simulator image:

ARCH=x86_64 ./tools/build.sh --p4_program sdn_policy_offload
The full syntax for the tools/build.sh command is:
# tools/build.sh --help
ARCH=<architecture> --p4_program <p4_program> [--clean] [--create-sim-
target]
Architectures: aarch64
x86_64 (default)
--p4_program: hello_world
flow_offload
classic_host_offload
classic_rtr flow_offload
flow_ha
sdn_policy_offload
sdn_policy_offload
--clean: cleaning target and build directories (optional)
--create-sim-target: build the rudra target sim
UG1668 (v1.0) November 30, 2023

Send Feedback
Managing the Simulation Environment

Once the build process is complete, start the simulator on the built reference pipeline using the
following command:
./tools/dsc-sim-start.sh --p4_program sdn_policy_offload
At this point both the ASIC simulator and core application are running, and the two uplink
interfaces (uplink0 and uplink1) are exposed to Linux® as tap interfaces (Eth1-1 and Eth1-2).
Validate this by running show commands such as the following:
pdsctl show lif

pdsctl show port status
As a part of the simulator startup, the dp-app process is started, which reads the file /sw/nic/
rudra/src/conf/sdn_policy_offload/config.json. This file contains instructions to
create the flow with policy and route results, and to populate specific entries into the tables. To
view the file, run the command:
cat /sw/nic/rudra/src/conf/sdn_policy_offload/config.json
If desired you can expose the uplink interfaces outside the container, so you can run traffic tools
from outside to inject packets to the uplinks, To do so, run the following commands on the host
of the container:
cd $TTOP/src/github.com/pensando/sw/nic/
DSC_INSTANCE=<ssdk-dev-container-name> ./rudra/test/tools/setup-uplink-
interfaces.sh
Note: <ssdk-dev-container-name> can be obtained via docker ps.
To stop the simulator environment, execute the dsc-sim-stop.sh script. This kills the
processes including the ASIC simulator and pds core application:
./tools/dsc-sim-stop.sh
To restart the simulation environment, execute the dsc-sim-restart.sh script. This restarts
all the processes including the ASIC simulator and pds core application, which is useful in
recovering from scenarios such as a process crashing:
./tools/dsc-sim-restart.sh --p4_program sdn_policy_offload
UG1668 (v1.0) November 30, 2023

Send Feedback
Building a DSC Firmware Image

To build a DSC firmware image, do the following within the P4 development container:

cd /sw/nic

ARCH=aarch64 ./tools/build.sh --p4_program sdn_policy_offload --clean
3. Build the DSC firmware image:

ARCH=aarch64 ./tools/build.sh --p4_program sdn_policy_offload
This creates an image for the DSC/DPU as a tarball in the /sw/nic directory, for example
dsc_fw_elba_.tar.
4. List the built DSC firmware image:
ls /sw/nic/dsc_fw_elba_.tar
Note: This is also aliased to naples_fw_elba.tar.
Copying an Image to the DSC

After building the DSC firmware image you must copy it to the DSC. You can use any of the
following in-band or out-of-band management interfaces:
• The internal management NIC (int_mnic) interface

• The memtun interface
• A serial cable connected to the DCS console
• The external RJ45 out-of-band (OOB) interface
Using the Internal Management NIC Interface

To copy the DSC firmware image using the internal management NIC (int_mnic) interface:
1. Boot up the host with the DSC plugged in.

2. Login to the host.
UG1668 (v1.0) November 30, 2023

Send Feedback
3. Confirm the DSC enumerates as a PCIe® device, and note the PCIe address of its
management controller:
# lspci -d 1dd8:
12:00.0 PCI bridge: Pensando Systems Inc Device 0002
13:00.0 PCI bridge: Pensando Systems Inc DSC Virtual Downstream Port
14:00.0 Ethernet controller: Pensando Systems Inc DSC Ethernet Controller
16:00.0 Ethernet controller: Pensando Systems Inc DSC Management
Controller
In this example the PCIe address is 16:00.0.

4. Load the IONIC Ethernet driver on the host.
5. Identify the interface for the DSC management controller.
Look in the dmesg logs from when you were loading the driver, find the PCIe address from
step 3 (16:00.0 in this example), and note the associated interface:
# dmesg | tail
[ 341.163138] ionic Pensando Ethernet NIC Driver, ver 23.02.1-001
[ 341.163334] ionic 0000:14:00.0: 126.016 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:11:00.0 (capable of
252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 341.165911] ionic 0000:14:00.0: FW: 1.51.0-G-37
[ 341.210301] ionic 0000:14:00.0 ens1: renamed from eth0
[ 341.212756] ionic 0000:15:00.0: FW: 1.51.0-G-37
[ 341.263733] ionic 0000:16:00.0: FW: 1.51.0-G-37
[ 341.345005] ionic 0000:16:00.0 eth1: Link up - 1 Gbps
The final line shows that the driver is using is using the eth1 interface for the DSC
management controller.
6. Configure an IP address for this interface:
# ifconfig eth1 169.254.<mgmt_ctlr_pcie_bus_number>.2/24 up
where:
• <mgmt_ctlr_pcie_bus_number> is the first component of the PCIe address for the
management controller from step 3, converted from hexadecimal to decimal.
In this example, the PCIe address is 16:00.0, so the bus address is 0x16 which converts
to 22 decimal.
The command corresponding to the example output shown in step 3 is therefore:
# ifconfig eth1 169.254.22.2/24 up
UG1668 (v1.0) November 30, 2023

Send Feedback
7. Confirm the IP address has been set:

# ifconfig eth1
ens1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 169.254.22.2 netmask 255.255.255.0 broadcast
169.254.22.255
ether 00:ae:cd:2e:70:d0 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
8. Verify that you can ping the DSC from the host:
Note: The internal management NIC for the DSC has a default IP address of 169.254.22.1.
# ping 169.254.22.1 -c 3
PING 169.254.22.1 (169.254.22.1) 56(84) bytes of data.
64 bytes from 169.254.22.1: icmp_seq=1 ttl=64 time=0.205 ms
--- 169.254.22.1 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2037ms
9. SSH to the DSC from the host as root with a password of pen123 :
# ssh -lroot 169.254.22.1
The authenticity of host '169.254.22.1 (169.254.22.1)' can't be
established.
ECDSA key fingerprint is SHA256:AoU0vi8BifouUOfqSg78t08JgaH7vHHBZfK58CnS
+EI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '169.254.22.1' (ECDSA) to the list of known
hosts.
root@169.254.22.1's password:
10. Confirm the IP address for the internal Management NIC (MNIC) interface on the DSC is
169.254.<pcie_bus_number>.1:
Note: The int_mnic0 interface is the internal MNIC net device on the DSC that is created by default.
# ifconfig int_mnic0
int_mnic0 Link encap:Ethernet HWaddr 00:AE:CD:01:C6:C4
inet addr:169.254.22.1 Bcast:169.254.22.255 Mask:255.255.255.0
inet6 addr: fe80::2ae:cdff:fe01:c6c4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:41 errors:0 dropped:0 overruns:0 frame:0
TX packets:65 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5351 (5.2 KiB) TX bytes:10066 (9.8 KiB)
11. Use scp to copy the image over the internal management NIC interface from the host
(specified by the IP address that you set in step 6) to the /data/ directory on the card:
$ scp <user>@<host_ip_address>:<SSDK_dir>/src/github.com/pensando/sw/nic/
dsc_fw_elba_.tar /data/
UG1668 (v1.0) November 30, 2023

Send Feedback
For example:
$ scp jdoe@169.1.22.2:/home/jdoe/ssdk/src/github.com/pensando/sw/nic/
Using the Memtun Interface

Memtun is another PCIe memory-based interface. It is available for Linux hosts, and does not
require the IONIC driver on the host, but is less secure.
To copy the DSC firmware image using the memtun interface:
1. Boot up the host with the DSC plugged in.

2. Login to the host.
3. Confirm the DSC enumerates as a PCIe device, and note the PCIe addresses of its first virtual
downstream port and of its management controller:
# lspci -d 1dd8:
12:00.0 PCI bridge: Pensando Systems Inc Device 0002
16:00.0 Ethernet controller: Pensando Systems Inc DSC Management
Controller
In this example the PCIe addresses are 13:00.0 and 16:00.0, respectively.
4. Compile the memtun binary for the host using the code from $sw/platform/src/app/
memtun.
5. Start the memtun binary on the host:
# ./memtun -s <downstream_port_pcie_address>
169.1.<mgmt_ctlr_pcie_bus_number>.2 &
where:
• <downstream_port_pcie_address> is the PCIe address of the virtual downstream
port from step 3.
In this example it is 13:00.0.
• <mgmt_ctlr_pcie_bus_number> is the first component of the PCIe address for the
management controller from step 3, converted from hexadecimal to decimal.
In this example, the PCIe address is 16:00.0, so the bus address is 0x16 which converts
to 22 decimal.
The command corresponding to the example output shown in step 3 is therefore:
# ./memtun -s 13:00.0 169.1.22.2 &
UG1668 (v1.0) November 30, 2023

Send Feedback
The memtun utility is started by default on DSCs running goldfw or ssdk firmware. When
memtun starts on the host it establishes a connection with the DSC and creates a tun device
visible in ifconfig on both the host and the DSC.
6. Confirm the IP address has been set:
# ifconfig tun0
tun0 Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:169.1.22.2 P-t-P:169.1.22.3
Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX bytes:96 (96.0 B) TX bytes:96 (96.0 B)
7. Verify that you can ping the DSC from the host:
Note: The tun device for the DSC has a default IP address of 169.254.22.3.
# ping 169.254.22.3 -c 3
PING 169.254.22.3 (169.254.22.3) 56(84) bytes of data.
--- 169.254.22.3 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2037ms
8. SSH to the DSC from the host as root with a password of pen123 :
# ssh -lroot 169.254.22.1
The authenticity of host '169.254.22.1 (169.254.22.1)' can't be
established.
ECDSA key fingerprint is SHA256:AoU0vi8BifouUOfqSg78t08JgaH7vHHBZfK58CnS
+EI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '169.254.22.1' (ECDSA) to the list of known
hosts.
root@169.254.22.1's password:
9. Confirm the IP address for the tun0 interface on the DSC is 169.254.<pcie_bus_number>.3:
Note: The tun0 interface is the memtun device on the DSC that is created by default.
# ifconfig tun0
tun0 Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:169.1.22.3 P-t-P:169.1.22.2
Mask:255.255.255.255
inet6 addr: fe80::b1a3:b97a:c6a3:3b12/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX bytes:96 (96.0 B) TX bytes:96 (96.0 B)
UG1668 (v1.0) November 30, 2023

Send Feedback
10. Use scp to copy the image over the memtun interface from the host (specified by the IP
address that you set in step 5) to the /data/ directory on the card:
$ scp <user>@<host_ip_address>:<SSDK_dir>/src/github.com/pensando/sw/nic/
For example:
$ scp jdoe@169.1.22.2:/home/jdoe/ssdk/src/github.com/pensando/sw/nic/
Using the DSC Console

To copy the DSC firmware image using the DSC console:
1. Connect the DSC console to the host with a serial cable.

2. Use scp to copy the image over the serial cable from the host to the /data/ directory on
the card.
$ scp <user>@<host>:<SSDK_dir>/src/github.com/pensando/sw/nic/
Using the External RJ45 OOB Interface

If the DSC has an external RJ45 OOB interface you can connect it to the management network,
configure it, and use it to copy images:
1. Ensure the oob_mnic0 interface is up:

ifconfig oob_mnic0
If you get no output:

• Run the following command:
dhclient oob_mnic0
• Boot `gold-fw` if needed.

2. Use scp to copy the image over the out of band interface to the /data directory on the
card:
scp user@server://~dsc_fw_elba.tar
Loading an Image
To load an image you have copied to the DSC, do the following on the DSC:
UG1668 (v1.0) November 30, 2023

Send Feedback
1. Synchronize data on disk with memory:

# sync
2. Set the copied image as the boot image:

# sysupdate.sh -p /data/dsc_fw_elba_.tar
3. Restart the DSC:

# sysreset.sh
The DSC comes up with the new firmware loaded.

4. Get the details of the loaded firmware and confirm that the version number and build time
are as expected:
# pdsctl show version
Firmware Version : <version_number>
Pipeline : rudra
P4 Program : sdn_policy_offload
Build Time : <build_time>
Note: You can also use the table access tools (p4ctl or, if one of the reference pipelines had been
used to create the image, pdsctl) to verify that the correct software is executed.
Starting the DP-App

The dp-app is not started by default in hardware. You must start it yourself.
To do so:
1. Start the dp-app as follows:

• To use the default init config from /nic/conf/config.json:
start-dp-app.sh
• To use a different config file specify it with the -c option. For example:
start-dp-app.sh -c /nic/config/config_scale.json
2. Use grep to confirm that the dp-app is running:

# ps -ef | grep pds_dp
root 1659 1 99 00:00 ttyS0 00:07:38 pds_dp_app -c /nic/conf/
config.json
root 1733 578 0 00:00 ttyS0 00:00:00 grep pds_dp
UG1668 (v1.0) November 30, 2023

Send Feedback
Cleaning the Build

You must clean the current build before doing any of the following:
• Building a P4 program for the first time.

• Building a different P4 program.
• Switching between x86 and Arm targets.
To do so:

cd /sw/nic

ARCH=x86_64 ./tools/build.sh --p4_program sdn_policy_offload --clean
Performing a Packet Test

After you have initialized the dp-app with the default config.json, you can perform packet
tests by sending packets to the exposed uplink interfaces (Eth1-1, Eth1-2).
A sample packet test is described at the following location:
/sw/nic/rudra/docs/sdn_policy_offload/quickstart.md#sim-packet-test
Troubleshooting
The following files in the container are useful for troubleshooting:
• Persistent logs files are located at /obfl/.

• Non-persistent log files are located at /var/log/pensando/.
• If you start the ASIC simulator in debug mode, a model.log file is generated in the non-
persistent log location (/var/log/pensando/).
• In the event of a primary application core dump, the core file is located at /data/core/.
• Technical Support files are generated at /data/techsupport/.
UG1668 (v1.0) November 30, 2023

Send Feedback
Appendix A: Additional Resources and Legal Notices
Appendix A
Additional Resources and Legal

Notices
Finding Additional Documentation

Documentation Portal
The AMD Documentation Hub is an online tool that provides robust search and navigation for
documentation using your web browser. To access the Documentation Hub, go to https://
www.amd.com/en/search/documentation/hub.html.
Support Resources
For support resources such as Answers, Documentation, Downloads, and Forums, see Support.
References
Pensando Documents
1. SSDK Product Brief (PPB22002)

2. Pensando DSC2-200 Data Sheet (PPB23003)
Revision History
The following table shows the revision history for this document.
UG1668 (v1.0) November 30, 2023

Send Feedback
Appendix A: Additional Resources and Legal Notices
Section Revision Summary

11/30/2023 Version 1.0
Initial release. N/A
Please Read: Important Legal Notices

The information presented in this document is for informational purposes only and may contain
technical inaccuracies, omissions, and typographical errors. The information contained herein is
subject to change and may be rendered inaccurate for many reasons, including but not limited to
product and roadmap changes, component and motherboard version changes, new model and/or
product releases, product differences between differing manufacturers, software changes, BIOS
flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities
that cannot be completely prevented or mitigated. AMD assumes no obligation to update or
otherwise correct or revise this information. However, AMD reserves the right to revise this
information and to make changes from time to time to the content hereof without obligation of
AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED "AS
IS." AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE
CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES,
ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY
DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR
FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY
PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL
DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF
AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Copyright
© Copyright 2023 Advanced Micro Devices, Inc. AMD, the AMD Arrow logo, Pensando, and
combinations thereof are trademarks of Advanced Micro Devices, Inc. AMBA, AMBA Designer,
Arm, ARM1176JZ-S, CoreSight, Cortex, PrimeCell, Mali, and MPCore are trademarks of Arm
Limited in the US and/or elsewhere. PCI, PCIe, and PCI Express are trademarks of PCI-SIG and
used under license. Other product names used in this publication are for identification purposes
only and may be trademarks of their respective companies.
UG1668 (v1.0) November 30, 2023

Send Feedback

Pensando SSDK SDN Policy Offload User Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pensando SSDK SDN Policy Offload User Guide

Uploaded by

Copyright:

Available Formats

SDN Policy Offload

UG1668 (v1.0) November 30, 2023

AMD Adaptive Computing is creating an environment where

Chapter 3: Building and Running the Reference Pipeline..................... 15

Appendix A: Additional Resources and Legal Notices............................. 26

UG1668 (v1.0) November 30, 2023

• Quickly develop your own networking software

UG1668 (v1.0) November 30, 2023

Benefits of using a software-defined data plane include:

• Performance: The AMD PensandoPensando DPU is purpose-built to offload and accelerate

SSDK SDN Policy Offload Pipeline Overview

UG1668 (v1.0) November 30, 2023

• Routing (LPM and Priority-Based Route Lookup).

○ Policy lookups in the P4-DMA data path.

UG1668 (v1.0) November 30, 2023

• Hardware Assisted Flow Aging

○ Testing Tool: Flow Creation: send a packet with a SCAPI script.

Build Your Own Libraries

• Flow offload (64 million sessions) with connection tracking.

UG1668 (v1.0) November 30, 2023

• Virtual Networking (Overlay)

○ Tenant aware routing (VRFs).

○ Network address translation for inter-VPC or out-bound traffic.

○ Network service mesh: Policy driven service functions chaining.

○ Custom routing via workload APIs or via BGP.

○ Overlay mapping tables.

○ Source RPF security mapping (SIP to TEP mapping).

○ Stateless network ACL tables.

UG1668 (v1.0) November 30, 2023

• Physical Network (Underlay) Integration

○ LLDP for topology awareness.

○ LACP for bonding up L2 to TORs.

○ Topology Map: Provides the workloads with TOR connectivity.

UG1668 (v1.0) November 30, 2023

UG1668 (v1.0) November 30, 2023

Figure 1: BITW Implementation for SmartSwitch Appliances

DPU ×N x86 System

Figure 2: Host to Network Implementation

x86 System pf0

SDN Policy Implementation

• SDN policy evaluation (LPM table lookup for forwarding decision)

UG1668 (v1.0) November 30, 2023

Figure 3: SDN Policy Offload API Model

AMD Library in P4 AMD Library in P4

Custom P4 Code Custom P4 Code

Eth 1/1 Eth 1/2

Uplink Port → Ingress (flow-miss) → RxDMA → TxDMA → Egress → RxDMA → DP App →

Uplink Port → Ingress (flow-hit) → Egress → Uplink Port

UG1668 (v1.0) November 30, 2023

Figure 4: Packet Flow-Miss and Flow-Hit Paths

Polls on Datapath LIF

LPM Based Routing LPM Based Routing

Idle Aging + Connection Tracking

Eth 1/1 Eth 1/2 SxDMA

Topology for Testing SDN Policy Offload

Single DSC Topology

UG1668 (v1.0) November 30, 2023

Figure 5: Single DSC Topology for Testing

AD-App SDN Policy Acceleration Reference

Eth 1/1 Eth 1/2

VXLAN Encapsulated IP Packet

Dual DSC Topology

UG1668 (v1.0) November 30, 2023