November 16, 2026
There and back again: a 1.74 µs tale with Backline
What happens to Backline's round trip latency if you change one block of logic on the FPGA? We added support for NVIDIA's open-source Holoscan Sensor Bridge, kept the same workstation, network card and cable, and the round trip latency from the FPGA to a CPU and back fell from 2.47 µs to 1.74 µs. This is, to our knowledge, the lowest published for quantum controllers between an FPGA and a CPU over a switched network protocol.
Time spent moving data is time the decoder can't use: the shorter the round trip latency, the more room there is for decoding work.
In September, together with AMD, we announced Backline, an open platform for compiling quantum programs and running them across FPGAs, CPUs, and GPUs with low latency, all from Python. In October, we showed how to bring your own decoder to it. Today, we're opening up the other end of the link: the RDMA engine inside the FPGA itself.
An RDMA engine is the block of FPGA logic that turns data into network packets and back. In addition to AMD ERNIC™ support, we've added NVIDIA's open-source Holoscan Sensor Bridge (HSB), and measured the two on the same AMD Ryzen™ Threadripper™ PRO workstation, with the same network card and the same cable.
With HSB, the round trip latency from the FPGA to the CPU and back takes 1.74 µs, down from 2.47 µs with ERNIC. To our knowledge, that's the lowest published round trip for quantum control between a CPU and an FPGA over a switched network protocol. To an AMD Instinct™ MI210 GPU, it fell from 4.56 µs to 3.75 µs.
Read on for what changed, how these numbers compare with other published results, and how to choose an RDMA engine from PennyLane.
What's new?
Backline connects the control FPGA sitting next to the quantum hardware with the CPUs and GPUs running your decoders, using RDMA over Converged Ethernet (RoCE v2).
On the FPGA, the piece of logic that speaks RoCE (pronounced 'rocky') is the RDMA engine. Until now, Backline has used AMD's ERNIC, a full-featured, queue-based RDMA engine that is already part of many AMD Versal designs. With this update, you can now also use NVIDIA's Holoscan Sensor Bridge. HSB comes from NVIDIA's Holoscan platform, where it streams camera, radar, and RF data from FPGAs directly into GPU memory. It's lightweight, open source under the Apache 2.0 license, and a natural fit for a stream of syndrome data. (If HSB sounds familiar, that might be because it's also the FPGA IP behind NVQLink, NVIDIA's reference architecture for connecting quantum control systems to GPUs.)
From PennyLane, switching to HSB takes two settings. Simply load the HSB bitstream (the FPGA
design) onto the board, pass backend="hsb_hwhs" to the controller, and add peer=hsb to the
coprocessor's config:
import pennylane as qp
FPGA = qp.Controller(
name="fpga-controller",
remote=True,
hardware="fpga",
backend="hsb_hwhs", # HSB; backend="hwhs" is ERNIC
executor_options={**FPGA_SERVER, "port": 9811},
init_args={"config": "reply_to_us=1000000"},
)
GPU = qp.Coprocessor(
name="gpu-coproc",
coprocessor_fn="gpu_steane_launcher",
remote=True,
hardware="gpu", # the Instinct MI210
endpoint=qp.Endpoint(WORKSTATION_ADDRESS, 7760), # an address the FPGA board can reach
executor_options={**WORKSTATION_SERVER, "port": 9813},
init_args={"config": "dev=mlx5_1;gid=5;peer=hsb"}, # HSB; peer=catalyst is ERNIC
)
Here, FPGA_SERVER and WORKSTATION_SERVER hold the connection details for your FPGA board and
workstation. Your QNodes, compiler passes, and decoders all stay the same.
To see what HSB brings to the table compared to our previous Backline benchmarking experiments, we kept everything else exactly the same — the same AMD Versal VPK120 FPGA, connected by a single cable to the same AMD Threadripper PRO workstation.
A new floor: 1.74 µs
On every workload we measured, HSB comes in 0.7 to 0.8 µs lower than ERNIC — both at the median, and out in the tail.
The CPU results are where HSB really shines. The median round trip latency to the CPU is just 1.74 µs, and the 99.9th percentile sits only 28 ns above it. The two RDMA engines don't overlap at all — across a million rounds of the CPU echo each, HSB's slowest round (1.96 µs) was still faster than ERNIC's fastest (2.45 µs)!
Including a decoder in the workflow doesn't change the picture. The Steane decoder is light enough that it adds nothing we could measure on the CPU, allowing HSB's savings to carry straight through. Even with a much heavier qLDPC decoder on the GPU, where decoding takes up most of the 32 µs round trip, HSB still saves 0.8 µs. This makes sense; the change is entirely on the FPGA side, so every workload benefits.
| Coprocessor | Workload | HSB median (µs) | ERNIC median (µs) | HSB 99.9th pct. (µs) | ERNIC 99.9th pct. (µs) |
|---|---|---|---|---|---|
| Threadripper PRO CPU | Echo | 1.736 | 2.470 | 1.764 | 2.610 |
| Threadripper PRO CPU | Steane decoder | 1.736 | 2.470 | 1.764 | 2.610 |
| Instinct MI210 GPU | Echo | 3.748 | 4.555 | 4.428 | 5.250 |
| Instinct MI210 GPU | Steane decoder | 3.852 | 4.550 | 4.588 | 5.285 |
| Instinct MI210 GPU | qLDPC decoder | 31.84 | 32.65 | 32.56 | 33.41 |
For each measurement, we ran a million rounds (dropping the first as a warm-up). Logic in the FPGA fabric issued each round as soon as the previous reply arrived; we call this a hardware loop, run back to back. The timer runs on the FPGA itself, from sending the request to receiving the reply, allowing it to cover the entire path: the RDMA engine in both directions, the cable, the network card, and the CPU or GPU. We also tried issuing a round every 10 µs with HSB, rather than back to back, and saw the same median to within a few nanoseconds.
Going back to the latency tiers from our September post, this puts both the CPU and the GPU in the synchronous co-processing tier (1 to 100 µs) used for tasks like backup QEC decoding — but now with an extra 0.7 to 0.8 µs of every round available to the decoder itself.
How does this compare?
To put these numbers in context, we went looking for every published round trip latencies we could find between a quantum control FPGA and a CPU or GPU, across papers, technical blogs, and vendor announcements.
But first, what do we mean by "over a switched network protocol"? Our FPGA is network-attached. It has its own network port, and talks to the host's network card using a protocol built to run through network switches (RoCE v2, in our case), rather than sitting on a PCIe card inside the host or sharing a chip with the CPU. Among published round trip latencies between an open protocol network-attached FPGA and a CPU for quantum control, 1.74 µs is, to our knowledge, the lowest.
Note that we measured over a single direct attach cable between the VPK120 and the ConnectX-7, with no switch in between, comparable to NVIDIA's NVQLink and DOCA GPUNetIO results. A real installation may add switches, and each one adds its own latency in both directions (THQLink reports 0.13 µs for each extra switch hop on its network).
For CPUs, the lowest result we found from anyone else is THQLink, which connects an FPGA to an Intel Xeon server over the Tianhe supercomputer's proprietary interconnect. THQLink reports a median of 2.87 µs, 1.1 µs above Backline with HSB. NVIDIA also ships a CPU path for HSB, but we haven't yet seen a round trip latency published for it.
For GPUs, it's a different story. NVIDIA reported 3.84 µs for NVQLink in October 2025, and our 3.75 µs to the MI210 is on par with that. Since then, however, NVIDIA has brought its own HSB-based round trip latency down to 2.7 µs with DOCA GPUNetIO and 2.92 µs with cudaq-realtime, both on Blackwell GPUs — about 1 µs lower than ours.
We have a good idea of where most of that gap comes from. In NVIDIA's setup, the GPU kernel posts each reply to the network card itself, while in ours, the GPU kernel hands its reply off to a CPU thread to post. We believe that the extra 2 µs our GPU path takes over the CPU path is mostly spent on the host side, between the network card and the GPU kernel and back. (The hardware differs too, of course.) Letting the MI210 post its own replies could close some of that gap, and swapping the GPU may also yield better performance as well.
Under the hood: queues versus streams
So why is HSB faster? The answer lies in how each RDMA engine expects to be handed work.
On the FPGA, Backline's own controller logic — the hardware handshake,
or hwhs — issues each round, numbers it, times it, and matches up the
reply when it arrives. It does the same job whether ERNIC or HSB sits
behind it, but isn’t identical. Each bitstream carries one RDMA engine,
along with a hardware handshake built to fit it. So where does
the difference come from?
-
ERNIC is queue-based, much like the RDMA network cards found in data-center servers. To send a message, the controller writes the payload into memory, builds a work request describing it, and rings a doorbell to let ERNIC know that work is waiting. ERNIC then fetches both from memory before it can build and send the packet. This approach is very general, and supports many connections, reliable delivery, and large transfers — which is what most RDMA applications need.
-
HSB, on the other hand, is a streaming bridge. The controller hands it a single 64-byte word, and HSB builds the RoCE packet immediately, with the reply coming back the same way. There are no work requests, no doorbells, and no memory reads on the FPGA side before anything is sent.
This shorter path is where most of the gain comes from. (HSB also runs at a faster clock on our board, 250 MHz compared to ERNIC's 200 MHz.)
It's worth noting that these design choices are intentional, and your RDMA engine of choice may differ depending on your needs. A queue-based design is the right choice for general-purpose RDMA; however for specific jobs — for example, sending a small syndrome as fast as possible — a streaming design removes overhead.
Best of all, you don't have to pick just one. The results above came from an AMD CPU, GPU, and FPGA, running NVIDIA's RDMA engine and network card — and mixing and matching like this is exactly what Backline was built for. Since everything else stays the same whichever RDMA engine sits on the FPGA, you can choose the one that best fits each experiment with a simple configuration change.
Which RDMA engine should I use?
Each RDMA engine makes a different trade-off, so you can pick the one that suits your experiment:
| If you need... | Use | Controller backend | Coprocessor peer |
|---|---|---|---|
| The shortest round trip over a direct attach cable | NVIDIA HSB (Apache 2.0) | hsb_hwhs | peer=hsb |
| Reliable connections or many producers, with AMD IP already in many Versal designs | AMD ERNIC | hwhs (default) | peer=catalyst (default) |
It's also worth knowing that HSB uses RDMA's unreliable connected mode, as NVQLink does. On a direct attach cable between the FPGA and the network card, this keeps acknowledgement traffic off the critical path. However, it also means there's no retransmission at the RDMA layer, and no RDMA reads. (HSB does use one-sided RDMA writes, though; that's how each request reaches the CPU or GPU in our setup.) If you need retransmission, ERNIC uses reliable connections.
What's next?
Changing just one block of logic on the FPGA brought the round trip latency to the CPU down from 2.47 µs to 1.74 µs — to our knowledge, the lowest published for quantum control between a CPU and an FPGA over a switched network protocol. To the GPU, it fell from 4.56 µs to 3.75 µs. That's 0.7 to 0.8 µs back in every single round, on every workload we measured, for the decoder to use.
But there's still plenty more to explore. On the GPU side, for example, having the GPU post its replies to the network card directly, rather than handing them to a CPU thread, could bring the round trip latency down even further. And with more of every round now free for decoding, we're excited to see which decoders you bring to Backline with our bring-your-own-decoder workflow.
Try it out and get involved
Ready to try HSB with Backline? Here's where to start:
-
Get the code: The HSB design for the VPK120, including its driver and the test coprocessors, is available in the backline-vpk120 repository. HSB itself comes from NVIDIA's Holoscan Sensor Bridge project.
-
Explore the technology: Read the Backline paper for a technical overview of the architecture and its original measurements.
-
New to Backline? Read the announcement, and see how to go from local CPU prototyping to low-latency FPGA-GPU communication in our demo. Then bring your own decoder.
Have an RDMA engine you'd like to see running with Backline? Let us know! Head over to our GitHub repository to check out the ongoing work and join in on the development, or pop over to the PennyLane discussion forum to tell us how you are using PennyLane, Backline, and Catalyst in your workflows.
And if you are as excited as we are, make sure to sign up for the PennyLane newsletter for the latest PennyLane features and updates!
About the authors
Hong-Sheng Zheng
Hong-Sheng Zheng is a compiler engineer on the Compilation team at Xanadu.
Ali Farazdaghi
Ali is a systems software developer focused on performance, low-latency communication, GPUs and compute accelerators.
Joseph Lee
Joseph is a physicist and software developer. At Xanadu, he leads the performance team and makes PennyLane go zoom.
Lee O'Riordan
Physicist, purveyor of angular momentum, GPUs, pointy guitars, and computational things. Working on quantum stuff.
Josh Izaac
Josh is a theoretical physicist, software tinkerer, and occasional baker. At Xanadu, he contributes to the development and growth of Xanadu’s open-source quantum software products.