John Mower, Senior Research Scientist Engineer, University of Washington Jeff Johnson, FPGA Consultant, Opsero
When systems need to move enormous volumes of data in real time, traditional CPU-centric networking quickly becomes a bottleneck.
As a result, there’s a move towards logic connected Ethernet, the enhanced high bandwidth capabilities of up to 100 Gigabits per second (Gbit/s) bringing control, efficiency, and architectural freedom as well as speed.
By moving packet handling, filtering, and even protocol logic into the FPGA fabric itself, we can reduce latency, maximize bandwidth utilization, tailor protocols to application-specific needs, and free CPU resources for other tasks.
That’s the aim, but how do you get there?
In the first part of this series, we’ll look at a 100G networking solution where the System-on-Chip (SoC) uses the same physical link as high-rate logic-based Ethernet. Here, a processor, Direct Memory Access (DMA) controller, and Media Access Controller (MAC) work together to allow for the filtering and insertion of User Datagram Protocol (UDP) packets. One reason to use this approach is to offload data when the required throughput such as in radar systems or large phased-array sonars exceeds what the processor can handle. Another, and the one we’re focusing on, is to enable a remote endpoint for high-rate processing via Remote Direct Memory Access (RDMA).
Our thinking
Previous work in the radar and sonar world led us to a novel approach to moving data between devices. Modern SoCs have the advantage of offering FPGA resources with easily-programmable control for software-defined systems. The common paradigm in time-series offloading is to use a DMA engine to transfer data into the Processing System (PS) memory (RAM), then leverage the SoC’s built-in MAC for network communication.
An alternative is to utilize a logic-based MAC and connect to a physical layer device through FPGA I/O. In this case, the situation is highly similar to the previous approach where data is transferred via DMA to/from RAM and networking uses a separate DMA path, which raises a question. Could the PS RAM and network stack be bypassed entirely when the MAC is implemented in the Programmable Logic (PL)?
The answer is yes. We can let the DMA-connected MAC be configured by the PS and insert/filter our own frames, while allowing the traffic to/from the PS to maintain priority. We’ve previously implemented this on other proprietary stacks using both Verilog and High-Level Synthesis (HLS) at 1G speeds.
Our current work targets a network protocol called RDMA over Converged Ethernet version 2 (RoCEv2). While there are other networking approaches like InfiniBand, RoCEv1 and iWARP, RoCEv2 is routable and operates on a more versatile set of lower-cost hardware, encapsulating InfiniBand transport over UDP/IPv4. Additionally, it allows for remote PS communication over the same link. RoCEv2 uses UDP as its Layer 4 transport and is identified by destination port 4791.
Our goal is that the Versal SoC will ultimately act as a 100G RDMA source/sink and simultaneously have its PS available over a common physical layer.
Our approach
We could construct this system on a separate layer-by-layer approach. However, as we only intend on applications using Ethernet II, IPv4, and UDP, all three protocols are wrapped up in one IP and handled by a Linux/bare metal driver. Figure 2 shows the RTL IP placed between the typical DMA, FIFO, and Ethernet IPs.
A packet designed for use on a local area network (LAN) will have a 42 byte header where the IPv4 length, IPv4 checksum, and the UDP length are set on the fly (Figure 3). The IP has 48 byte wide S_TX_ARB and M_RX_STP AXI-Stream in/out ports and the _tuser field is used to signal the expected number of bytes in the frame. The other AXI-Stream ports are nominally 48 byte, however, an 8-byte wrapper is used for 10G systems where data width converters are used for compatibility. The reason to choose a nominal 48 byte interface is in anticipation of using the Versal 100G Multi Rate Media Access Controller (MRMAC) IP. When the IP is not active with new data in/out, DMA Ethernet frames are passed from S_RX -> M_RX and S_TX -> M_TX.
Receiving data
For the receive path, we simply filter incoming frames for protocol as well as desired destination MAC, IP, and port values. The filtering procedure simply delays the frame and decides whether the DMA or our internal IP has _tvalid asserted. What do we do about corrupted packets? The AMD 1G/2.5G, 10G/25G, and 100G MRMAC IPs all use a _tuser or extra _tkeep signal on the last beat to signal a frame check sequence error. A tail-drop FIFO is used on the filtered path to drop frames with an error. This we follow with a module that strips the headers, outputting a typical AXI-Stream frame with the number of bytes in _tuser.
The RX module is designed to operate without asserting backpressure and is capable of accepting continuous data from the MAC.
Transmitting data
The transmit path is a little more complicated, partially due to using backpressure for backwards compatibility. TX frames from the processor via DMA always have priority so a robust control system is possible. In reality, the processor will only ever occupy less than two percent of the 100G bandwidth and will not have a large impact on RDMA performance.
When a valid AXI-Stream frame is presented, we insert protocol headers and compute three values: IPv4 length (data + 28), UDP length (data + 8), and the IPv4 checksum. The IPv4 checksum is the one’s complement of the sum of each of the two-byte components of the header, with the carry added back in. Including the carry, this is ten words to add.
In our case we can simplify (and reduce required clock cycles) by realizing that we only intend to control the length, source address, and destination address fields in the IPv4 header. We can precompute a constant to create a smaller dedicated adder tree for the checksum, and the sum of the static header fields used in the IPv4 header in Figure 3 for our scenario is 0xc52d.
Linux
For Linux, we use a combined platform-character driver that uses typical open, close, and IOCTL system calls. The IOCTL system call is used to set the RX and TX parameters and run state as well as report statistics. While a driver could automatically set some of the fields, we leave it to the user to determine the proper local MAC, IP, and port addresses. It could be the case that we wish the logic region to have different values from the processor and it’s trivial to resolve peer MAC addresses using the ARP utility.
Device Tree note: Petalinux’s automatic pl.dtsi generation usually enables seamless networking. However, inserting an IP between DMA and Ethernet subsystem blocks in Vivado IP Integrator disrupts Device Tree generation. Certain fields must be removed or overwritten for proper Ethernet driver operation.
Testing
For testing, the M_RX_STP and S_TX_ARB ports are connected through a FIFO and therefore configured to work as a loopback server. To make it a bit whimsical, let’s send an ASCII string through the filter and arbiter and pick the string “I am a Cider Drinker” as a homage to a 1970s UK hit song and to what beverage is at hand right now. Figure 4 demonstrates the RX input on S_RX from the MAC and the M_RX_STP output (via a single byte width converter for readability).
Conclusion
In this first article we touch on a couple of topics related to implementing a Versal RDMA endpoint on a shared physical layer and in more general terms, UDP filtering and arbitration for DMA-connected Ethernet.
This project will be made available for the next installment at fpgadeveloper.com but in the meantime any requests can be directed to John Mower at mowerj@uw.edu.
